2023-12-16T21:42:56.806 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-16T21:42:56.807 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888 --name yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi --owner scheduled_yuriw@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} --verbose -- /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888/orig.config.yaml 2023-12-16T21:42:56.850 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T21:42:56.932 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888 branch: wip-yuri8-testing-2023-12-15-0911 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} email: yweinste@redhat.com first_in_suite: false job_id: '7494888' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri8-testing-2023-12-15-0911 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-12-15-0911 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2960 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2023-12-15-0911 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHqnBEcUc0TyMy+JYxM1Vd8gvT1HQU6hovtByw5v0ToGSNJ0XNATQJQEQsRRZ3thjojEItyZw/kqsayZ20D4rSw= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJ6URhJF7Vp3S0yxJdaNa9SgfnASuuzK9ws0P2U9Q88JHdZmDOM9uOsQdZ8pzetsGdReJKRBENC7VFvkslvzw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.tests_from_xfstests_dev teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-16_15:45:14 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-16T21:42:56.932 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa; will attempt to use it 2023-12-16T21:42:56.933 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks 2023-12-16T21:42:56.933 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-16T21:42:56.934 INFO:teuthology.task.internal:Checking packages... 2023-12-16T21:42:56.959 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c7586e95519b7be76f3b6e3a4b333888baf46bc9' 2023-12-16T21:42:56.959 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:42:56.959 INFO:teuthology.packaging:ref: None 2023-12-16T21:42:56.959 INFO:teuthology.packaging:tag: None 2023-12-16T21:42:56.959 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-12-15-0911 2023-12-16T21:42:56.959 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:42:56.960 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri8-testing-2023-12-15-0911 2023-12-16T21:42:57.179 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-202-gc7586e95-1jammy 2023-12-16T21:42:57.181 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-16T21:42:57.187 INFO:teuthology.task.internal:no buildpackages task found 2023-12-16T21:42:57.187 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-16T21:42:57.200 INFO:teuthology.task.internal:Saving configuration 2023-12-16T21:42:57.225 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-16T21:42:57.230 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T21:42:57.257 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 21:35:49.950107', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJpqMxXPssdx5OVczWsv8KkRJygwpiJ+YGgwfktrhATw'} 2023-12-16T21:42:57.280 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 21:35:49.948581', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIABKmCPf94qBzgU9wgq0RMNISi/yLolnkDQ2KvAXCje0'} 2023-12-16T21:42:57.280 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-16T21:42:57.286 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-12-16T21:42:57.286 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-12-16T21:42:57.286 INFO:teuthology.run_tasks:Running task console_log... 2023-12-16T21:42:57.360 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f32c4386ca0>, signals=[15]) 2023-12-16T21:42:57.360 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-16T21:42:57.366 INFO:teuthology.task.internal:Opening connections... 2023-12-16T21:42:57.366 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2023-12-16T21:42:57.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:42:57.445 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2023-12-16T21:42:57.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:42:57.522 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-16T21:42:57.529 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2023-12-16T21:42:57.534 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2023-12-16T21:42:57.535 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2023-12-16T21:42:57.580 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="Ubuntu" 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="22.04" 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_CODENAME=jammy 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:ID=ubuntu 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE=debian 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-16T21:42:57.581 INFO:teuthology.orchestra.run.smithi028.stdout:UBUNTU_CODENAME=jammy 2023-12-16T21:42:57.582 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2023-12-16T21:42:57.611 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2023-12-16T21:42:57.617 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2023-12-16T21:42:57.617 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Ubuntu" 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="22.04" 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_CODENAME=jammy 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:ID=ubuntu 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE=debian 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-16T21:42:57.669 INFO:teuthology.orchestra.run.smithi122.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-16T21:42:57.670 INFO:teuthology.orchestra.run.smithi122.stdout:UBUNTU_CODENAME=jammy 2023-12-16T21:42:57.670 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2023-12-16T21:42:57.695 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-16T21:42:57.704 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-16T21:42:57.710 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-16T21:42:57.710 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2023-12-16T21:42:57.712 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2023-12-16T21:42:57.717 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-16T21:42:57.722 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-16T21:42:57.722 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2023-12-16T21:42:57.758 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2023-12-16T21:42:57.787 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-16T21:42:57.843 INFO:teuthology.run_tasks:Running task kernel... 2023-12-16T21:42:57.857 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-16T21:42:57.857 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-16T21:42:57.858 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T21:42:57.858 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T21:42:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:42:57.863 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2023-12-16T21:42:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:42:57.864 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2023-12-16T21:42:57.909 INFO:teuthology.orchestra.run.smithi028.stdout:5.15.0-56-generic 2023-12-16T21:42:57.909 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.15.0-56-generic 2023-12-16T21:42:57.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get clean 2023-12-16T21:42:57.912 INFO:teuthology.orchestra.run.smithi122.stdout:5.15.0-56-generic 2023-12-16T21:42:57.912 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.15.0-56-generic 2023-12-16T21:42:57.912 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get clean 2023-12-16T21:42:58.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get update 2023-12-16T21:42:58.050 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2023-12-16T21:42:58.173 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-16T21:42:58.184 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-16T21:42:58.247 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-16T21:42:58.284 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-16T21:42:58.350 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-16T21:42:58.351 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-16T21:42:58.400 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-16T21:42:58.420 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [547 kB] 2023-12-16T21:42:58.485 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,263 kB] 2023-12-16T21:42:58.514 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [260 kB] 2023-12-16T21:42:58.518 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-16T21:42:58.518 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,250 kB] 2023-12-16T21:42:58.530 INFO:teuthology.orchestra.run.smithi028.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-16T21:42:58.530 INFO:teuthology.orchestra.run.smithi028.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [203 kB] 2023-12-16T21:42:58.532 INFO:teuthology.orchestra.run.smithi028.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-16T21:42:58.532 INFO:teuthology.orchestra.run.smithi028.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [675 kB] 2023-12-16T21:42:58.537 INFO:teuthology.orchestra.run.smithi028.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,020 kB] 2023-12-16T21:42:58.544 INFO:teuthology.orchestra.run.smithi028.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [226 kB] 2023-12-16T21:42:58.546 INFO:teuthology.orchestra.run.smithi028.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-16T21:42:58.548 INFO:teuthology.orchestra.run.smithi028.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-16T21:42:58.571 INFO:teuthology.orchestra.run.smithi028.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-16T21:42:58.572 INFO:teuthology.orchestra.run.smithi028.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-16T21:42:58.572 INFO:teuthology.orchestra.run.smithi028.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-16T21:42:58.572 INFO:teuthology.orchestra.run.smithi028.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-12-16T21:42:58.573 INFO:teuthology.orchestra.run.smithi028.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-12-16T21:42:58.573 INFO:teuthology.orchestra.run.smithi028.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-16T21:42:58.573 INFO:teuthology.orchestra.run.smithi028.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-16T21:42:58.573 INFO:teuthology.orchestra.run.smithi028.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-12-16T21:42:58.573 INFO:teuthology.orchestra.run.smithi028.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-12-16T21:42:58.589 INFO:teuthology.orchestra.run.smithi028.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-16T21:42:58.605 INFO:teuthology.orchestra.run.smithi028.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-16T21:42:58.721 INFO:teuthology.orchestra.run.smithi028.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [383 kB] 2023-12-16T21:42:58.755 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,051 kB] 2023-12-16T21:42:58.763 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-16T21:42:58.901 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [547 kB] 2023-12-16T21:42:58.988 INFO:teuthology.orchestra.run.smithi028.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,051 kB] 2023-12-16T21:42:59.090 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,263 kB] 2023-12-16T21:42:59.142 INFO:teuthology.orchestra.run.smithi028.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [200 kB] 2023-12-16T21:42:59.143 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [383 kB] 2023-12-16T21:42:59.158 INFO:teuthology.orchestra.run.smithi028.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-16T21:42:59.159 INFO:teuthology.orchestra.run.smithi028.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,226 kB] 2023-12-16T21:42:59.163 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [200 kB] 2023-12-16T21:42:59.174 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-16T21:42:59.174 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,226 kB] 2023-12-16T21:42:59.226 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [260 kB] 2023-12-16T21:42:59.232 INFO:teuthology.orchestra.run.smithi028.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-16T21:42:59.233 INFO:teuthology.orchestra.run.smithi028.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [200 kB] 2023-12-16T21:42:59.242 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-16T21:42:59.242 INFO:teuthology.orchestra.run.smithi028.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-16T21:42:59.243 INFO:teuthology.orchestra.run.smithi028.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [823 kB] 2023-12-16T21:42:59.243 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-16T21:42:59.266 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,250 kB] 2023-12-16T21:42:59.266 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-16T21:42:59.267 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [200 kB] 2023-12-16T21:42:59.270 INFO:teuthology.orchestra.run.smithi028.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [580 kB] 2023-12-16T21:42:59.274 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-16T21:42:59.275 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [580 kB] 2023-12-16T21:42:59.299 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [203 kB] 2023-12-16T21:42:59.300 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [823 kB] 2023-12-16T21:42:59.309 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-16T21:42:59.450 INFO:teuthology.orchestra.run.smithi028.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [156 kB] 2023-12-16T21:42:59.470 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,020 kB] 2023-12-16T21:42:59.487 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [156 kB] 2023-12-16T21:42:59.850 INFO:teuthology.orchestra.run.smithi028.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-16T21:42:59.856 INFO:teuthology.orchestra.run.smithi028.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-16T21:42:59.856 INFO:teuthology.orchestra.run.smithi028.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-16T21:42:59.870 INFO:teuthology.orchestra.run.smithi028.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-16T21:42:59.873 INFO:teuthology.orchestra.run.smithi028.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-16T21:42:59.881 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-16T21:42:59.888 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-16T21:42:59.888 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-16T21:42:59.902 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-16T21:42:59.905 INFO:teuthology.orchestra.run.smithi122.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-16T21:43:00.094 INFO:teuthology.orchestra.run.smithi122.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [675 kB] 2023-12-16T21:43:00.132 INFO:teuthology.orchestra.run.smithi122.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [226 kB] 2023-12-16T21:43:00.145 INFO:teuthology.orchestra.run.smithi122.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-16T21:43:00.146 INFO:teuthology.orchestra.run.smithi122.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-16T21:43:00.149 INFO:teuthology.orchestra.run.smithi122.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-16T21:43:00.149 INFO:teuthology.orchestra.run.smithi122.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-16T21:43:00.150 INFO:teuthology.orchestra.run.smithi122.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-16T21:43:00.150 INFO:teuthology.orchestra.run.smithi122.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-12-16T21:43:00.152 INFO:teuthology.orchestra.run.smithi122.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-12-16T21:43:00.177 INFO:teuthology.orchestra.run.smithi122.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-16T21:43:00.341 INFO:teuthology.orchestra.run.smithi122.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-16T21:43:00.450 INFO:teuthology.orchestra.run.smithi122.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-12-16T21:43:00.554 INFO:teuthology.orchestra.run.smithi122.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-12-16T21:43:00.582 INFO:teuthology.orchestra.run.smithi122.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-16T21:43:00.633 INFO:teuthology.orchestra.run.smithi122.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-16T21:43:04.338 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 10.8 MB in 2s (6,077 kB/s) 2023-12-16T21:43:05.009 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 10.8 MB in 2s (4,311 kB/s) 2023-12-16T21:43:05.579 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2023-12-16T21:43:05.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-16T21:43:05.657 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2023-12-16T21:43:05.849 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2023-12-16T21:43:05.850 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2023-12-16T21:43:06.020 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:43:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-16T21:43:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-16T21:43:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-16T21:43:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-16T21:43:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: python2.7-minimal 2023-12-16T21:43:06.021 INFO:teuthology.orchestra.run.smithi028.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:43:06.022 INFO:teuthology.orchestra.run.smithi028.stdout:The following additional packages will be installed: 2023-12-16T21:43:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: linux-generic linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-16T21:43:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: linux-headers-generic linux-image-5.15.0-91-generic 2023-12-16T21:43:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic 2023-12-16T21:43:06.023 INFO:teuthology.orchestra.run.smithi028.stdout:Suggested packages: 2023-12-16T21:43:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-16T21:43:06.023 INFO:teuthology.orchestra.run.smithi028.stdout:Recommended packages: 2023-12-16T21:43:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: thermald 2023-12-16T21:43:06.074 INFO:teuthology.orchestra.run.smithi028.stdout:The following NEW packages will be installed: 2023-12-16T21:43:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-16T21:43:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: linux-image-5.15.0-91-generic linux-modules-5.15.0-91-generic 2023-12-16T21:43:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: linux-modules-extra-5.15.0-91-generic 2023-12-16T21:43:06.075 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages will be upgraded: 2023-12-16T21:43:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-16T21:43:06.129 INFO:teuthology.orchestra.run.smithi028.stdout:3 upgraded, 5 newly installed, 0 to remove and 277 not upgraded. 2023-12-16T21:43:06.129 INFO:teuthology.orchestra.run.smithi028.stdout:Need to get 113 MB of archives. 2023-12-16T21:43:06.129 INFO:teuthology.orchestra.run.smithi028.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-16T21:43:06.129 INFO:teuthology.orchestra.run.smithi028.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101 [22.5 MB] 2023-12-16T21:43:06.465 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2023-12-16T21:43:06.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-16T21:43:06.537 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2023-12-16T21:43:06.578 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101 [11.5 MB] 2023-12-16T21:43:06.702 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101 [63.8 MB] 2023-12-16T21:43:06.704 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2023-12-16T21:43:06.705 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2023-12-16T21:43:06.847 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:43:06.848 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-16T21:43:06.848 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-16T21:43:06.848 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-16T21:43:06.848 INFO:teuthology.orchestra.run.smithi122.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-16T21:43:06.848 INFO:teuthology.orchestra.run.smithi122.stdout: python2.7-minimal 2023-12-16T21:43:06.849 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:43:06.849 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2023-12-16T21:43:06.849 INFO:teuthology.orchestra.run.smithi122.stdout: linux-generic linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-16T21:43:06.849 INFO:teuthology.orchestra.run.smithi122.stdout: linux-headers-generic linux-image-5.15.0-91-generic 2023-12-16T21:43:06.850 INFO:teuthology.orchestra.run.smithi122.stdout: linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic 2023-12-16T21:43:06.851 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2023-12-16T21:43:06.851 INFO:teuthology.orchestra.run.smithi122.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-16T21:43:06.851 INFO:teuthology.orchestra.run.smithi122.stdout:Recommended packages: 2023-12-16T21:43:06.851 INFO:teuthology.orchestra.run.smithi122.stdout: thermald 2023-12-16T21:43:06.898 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2023-12-16T21:43:06.899 INFO:teuthology.orchestra.run.smithi122.stdout: linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-16T21:43:06.899 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.15.0-91-generic linux-modules-5.15.0-91-generic 2023-12-16T21:43:06.899 INFO:teuthology.orchestra.run.smithi122.stdout: linux-modules-extra-5.15.0-91-generic 2023-12-16T21:43:06.900 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages will be upgraded: 2023-12-16T21:43:06.900 INFO:teuthology.orchestra.run.smithi122.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-16T21:43:06.952 INFO:teuthology.orchestra.run.smithi122.stdout:3 upgraded, 5 newly installed, 0 to remove and 277 not upgraded. 2023-12-16T21:43:06.952 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 113 MB of archives. 2023-12-16T21:43:06.952 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-16T21:43:06.952 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101 [22.5 MB] 2023-12-16T21:43:07.350 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101 [11.5 MB] 2023-12-16T21:43:07.449 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101 [63.8 MB] 2023-12-16T21:43:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.91.88 [1,700 B] 2023-12-16T21:43:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.91.88 [2,488 B] 2023-12-16T21:43:07.527 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91 all 5.15.0-91.101 [12.3 MB] 2023-12-16T21:43:07.655 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91-generic amd64 5.15.0-91.101 [2,860 kB] 2023-12-16T21:43:07.678 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.91.88 [2,342 B] 2023-12-16T21:43:08.099 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 113 MB in 2s (70.8 MB/s) 2023-12-16T21:43:08.245 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.91.88 [1,700 B] 2023-12-16T21:43:08.246 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.91.88 [2,488 B] 2023-12-16T21:43:08.246 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91 all 5.15.0-91.101 [12.3 MB] 2023-12-16T21:43:08.272 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-modules-5.15.0-91-generic. 2023-12-16T21:43:08.367 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91-generic amd64 5.15.0-91.101 [2,860 kB] 2023-12-16T21:43:08.474 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.91.88 [2,342 B] 2023-12-16T21:43:08.858 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 113 MB in 2s (72.1 MB/s) 2023-12-16T21:43:09.029 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-modules-5.15.0-91-generic. 2023-12-16T21:43:10.251 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-16T21:43:10.255 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../0-linux-modules-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-16T21:43:10.311 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:11.088 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-16T21:43:11.092 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../0-linux-modules-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-16T21:43:11.202 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:12.371 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-image-5.15.0-91-generic. 2023-12-16T21:43:12.392 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../1-linux-image-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-16T21:43:12.469 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:12.958 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-91-generic. 2023-12-16T21:43:12.979 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-16T21:43:13.016 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:13.529 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-image-5.15.0-91-generic. 2023-12-16T21:43:13.550 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../1-linux-image-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-16T21:43:13.626 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:14.074 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-91-generic. 2023-12-16T21:43:14.094 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-16T21:43:14.132 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:19.625 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../3-linux-generic_5.15.0.91.88_amd64.deb ... 2023-12-16T21:43:19.752 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-16T21:43:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.91.88_amd64.deb ... 2023-12-16T21:43:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-image-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-16T21:43:20.499 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-headers-5.15.0-91. 2023-12-16T21:43:20.522 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../5-linux-headers-5.15.0-91_5.15.0-91.101_all.deb ... 2023-12-16T21:43:20.558 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-16T21:43:21.308 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../3-linux-generic_5.15.0.91.88_amd64.deb ... 2023-12-16T21:43:21.427 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-16T21:43:21.768 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.91.88_amd64.deb ... 2023-12-16T21:43:21.897 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-image-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-16T21:43:22.234 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-headers-5.15.0-91. 2023-12-16T21:43:22.256 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../5-linux-headers-5.15.0-91_5.15.0-91.101_all.deb ... 2023-12-16T21:43:22.300 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-16T21:43:26.505 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package linux-headers-5.15.0-91-generic. 2023-12-16T21:43:26.542 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../6-linux-headers-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-16T21:43:26.580 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:28.322 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-headers-5.15.0-91-generic. 2023-12-16T21:43:28.359 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../6-linux-headers-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-16T21:43:28.396 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:29.248 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.91.88_amd64.deb ... 2023-12-16T21:43:29.374 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking linux-headers-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-16T21:43:29.832 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-16T21:43:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-headers-generic (5.15.0.91.88) ... 2023-12-16T21:43:30.208 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:31.125 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.91.88_amd64.deb ... 2023-12-16T21:43:31.258 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-16T21:43:31.550 INFO:teuthology.orchestra.run.smithi028.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-16T21:43:31.550 INFO:teuthology.orchestra.run.smithi028.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-16T21:43:31.550 INFO:teuthology.orchestra.run.smithi028.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-91-generic 2023-12-16T21:43:31.550 INFO:teuthology.orchestra.run.smithi028.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-91-generic 2023-12-16T21:43:31.617 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-16T21:43:31.709 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:31.738 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:31.856 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-generic (5.15.0.91.88) ... 2023-12-16T21:43:31.993 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:33.110 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-image-generic (5.15.0.91.88) ... 2023-12-16T21:43:33.210 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:33.300 INFO:teuthology.orchestra.run.smithi122.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-16T21:43:33.301 INFO:teuthology.orchestra.run.smithi122.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-16T21:43:33.301 INFO:teuthology.orchestra.run.smithi122.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-91-generic 2023-12-16T21:43:33.301 INFO:teuthology.orchestra.run.smithi122.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-91-generic 2023-12-16T21:43:33.495 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:34.629 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up linux-generic (5.15.0.91.88) ... 2023-12-16T21:43:34.755 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:34.824 INFO:teuthology.orchestra.run.smithi028.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-16T21:43:34.824 INFO:teuthology.orchestra.run.smithi028.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2023-12-16T21:43:34.905 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-image-generic (5.15.0.91.88) ... 2023-12-16T21:43:35.030 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:36.407 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-generic (5.15.0.91.88) ... 2023-12-16T21:43:36.534 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-16T21:43:36.598 INFO:teuthology.orchestra.run.smithi122.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-16T21:43:36.599 INFO:teuthology.orchestra.run.smithi122.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2023-12-16T21:43:49.902 INFO:teuthology.orchestra.run.smithi028.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-16T21:43:49.902 INFO:teuthology.orchestra.run.smithi028.stdout:Sourcing file `/etc/default/grub' 2023-12-16T21:43:49.918 INFO:teuthology.orchestra.run.smithi028.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-16T21:43:49.943 INFO:teuthology.orchestra.run.smithi028.stdout:Generating grub configuration file ... 2023-12-16T21:43:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-16T21:43:50.227 INFO:teuthology.orchestra.run.smithi028.stdout:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-16T21:43:50.533 INFO:teuthology.orchestra.run.smithi028.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-16T21:43:50.540 INFO:teuthology.orchestra.run.smithi028.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-16T21:43:50.618 INFO:teuthology.orchestra.run.smithi028.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-16T21:43:50.623 INFO:teuthology.orchestra.run.smithi028.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-16T21:43:50.820 INFO:teuthology.orchestra.run.smithi028.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-16T21:43:50.820 INFO:teuthology.orchestra.run.smithi028.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-16T21:43:50.820 INFO:teuthology.orchestra.run.smithi028.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-16T21:43:50.854 INFO:teuthology.orchestra.run.smithi028.stdout:done 2023-12-16T21:43:50.912 INFO:teuthology.orchestra.run.smithi122.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-16T21:43:50.913 INFO:teuthology.orchestra.run.smithi122.stdout:Sourcing file `/etc/default/grub' 2023-12-16T21:43:50.934 INFO:teuthology.orchestra.run.smithi122.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-16T21:43:50.959 INFO:teuthology.orchestra.run.smithi122.stdout:Generating grub configuration file ... 2023-12-16T21:43:51.283 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-16T21:43:51.304 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-16T21:43:51.329 DEBUG:teuthology.orchestra.run.smithi028:> dpkg -s linux-image-generic 2023-12-16T21:43:51.353 INFO:teuthology.orchestra.run.smithi028.stdout:Package: linux-image-generic 2023-12-16T21:43:51.353 INFO:teuthology.orchestra.run.smithi028.stdout:Status: install ok installed 2023-12-16T21:43:51.353 INFO:teuthology.orchestra.run.smithi028.stdout:Priority: optional 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Section: kernel 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Installed-Size: 20 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Maintainer: Ubuntu Kernel Team 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: amd64 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Source: linux-meta 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Version: 5.15.0.91.88 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Recommends: thermald 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout:Description: Generic Linux kernel image 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout: This package will always depend on the latest generic kernel image 2023-12-16T21:43:51.354 INFO:teuthology.orchestra.run.smithi028.stdout: available. 2023-12-16T21:43:51.354 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-91-generic 2023-12-16T21:43:51.355 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T21:43:51.355 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T21:43:51.355 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-16T21:43:51.355 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T21:43:51.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get clean 2023-12-16T21:43:51.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get update 2023-12-16T21:43:51.593 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-16T21:43:51.597 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-16T21:43:51.661 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-16T21:43:51.665 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-16T21:43:51.669 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-16T21:43:51.671 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-16T21:43:51.686 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-16T21:43:51.780 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-16T21:43:51.834 INFO:teuthology.orchestra.run.smithi122.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-16T21:43:51.834 INFO:teuthology.orchestra.run.smithi122.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-16T21:43:51.834 INFO:teuthology.orchestra.run.smithi122.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-16T21:43:51.867 INFO:teuthology.orchestra.run.smithi122.stdout:done 2023-12-16T21:43:52.400 DEBUG:teuthology.orchestra.run.smithi122:> dpkg -s linux-image-generic 2023-12-16T21:43:52.416 INFO:teuthology.orchestra.run.smithi122.stdout:Package: linux-image-generic 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Status: install ok installed 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Priority: optional 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Section: kernel 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Installed-Size: 20 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Maintainer: Ubuntu Kernel Team 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: amd64 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Source: linux-meta 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Version: 5.15.0.91.88 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Recommends: thermald 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout:Description: Generic Linux kernel image 2023-12-16T21:43:52.417 INFO:teuthology.orchestra.run.smithi122.stdout: This package will always depend on the latest generic kernel image 2023-12-16T21:43:52.418 INFO:teuthology.orchestra.run.smithi122.stdout: available. 2023-12-16T21:43:52.418 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-91-generic 2023-12-16T21:43:52.418 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T21:43:52.418 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T21:43:52.418 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-16T21:43:52.418 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T21:43:52.418 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get clean 2023-12-16T21:43:52.598 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2023-12-16T21:43:52.707 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-16T21:43:52.832 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-16T21:43:52.913 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-16T21:43:52.994 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-16T21:43:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2023-12-16T21:43:53.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-16T21:43:53.127 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2023-12-16T21:43:53.318 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2023-12-16T21:43:53.319 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2023-12-16T21:43:53.495 INFO:teuthology.orchestra.run.smithi028.stdout:linux-image-generic is already the newest version (5.15.0.91.88). 2023-12-16T21:43:53.495 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:43:53.496 INFO:teuthology.orchestra.run.smithi028.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-16T21:43:53.496 INFO:teuthology.orchestra.run.smithi028.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-16T21:43:53.496 INFO:teuthology.orchestra.run.smithi028.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-16T21:43:53.496 INFO:teuthology.orchestra.run.smithi028.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-16T21:43:53.496 INFO:teuthology.orchestra.run.smithi028.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-16T21:43:53.496 INFO:teuthology.orchestra.run.smithi028.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:43:53.537 INFO:teuthology.orchestra.run.smithi028.stdout:0 upgraded, 0 newly installed, 0 to remove and 277 not upgraded. 2023-12-16T21:43:53.538 DEBUG:teuthology.orchestra.run.smithi028:> dpkg -s linux-image-generic 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Package: linux-image-generic 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Status: install ok installed 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Priority: optional 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Section: kernel 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Installed-Size: 20 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Maintainer: Ubuntu Kernel Team 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: amd64 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Source: linux-meta 2023-12-16T21:43:53.552 INFO:teuthology.orchestra.run.smithi028.stdout:Version: 5.15.0.91.88 2023-12-16T21:43:53.553 INFO:teuthology.orchestra.run.smithi028.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-16T21:43:53.553 INFO:teuthology.orchestra.run.smithi028.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-16T21:43:53.553 INFO:teuthology.orchestra.run.smithi028.stdout:Recommends: thermald 2023-12-16T21:43:53.553 INFO:teuthology.orchestra.run.smithi028.stdout:Description: Generic Linux kernel image 2023-12-16T21:43:53.553 INFO:teuthology.orchestra.run.smithi028.stdout: This package will always depend on the latest generic kernel image 2023-12-16T21:43:53.553 INFO:teuthology.orchestra.run.smithi028.stdout: available. 2023-12-16T21:43:53.553 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2023-12-16T21:43:53.600 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.HY4vErvobU 2023-12-16T21:43:53.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub/grub.cfg /tmp/tmp.HY4vErvobU 2023-12-16T21:43:53.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.HY4vErvobU 2023-12-16T21:43:53.771 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.HY4vErvobU is 10KB 2023-12-16T21:43:53.784 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.HY4vErvobU 2023-12-16T21:43:53.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-16T21:43:53.845 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:43:53.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-16T21:43:53.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-16T21:43:53.911 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-91-generic 2023-12-16T21:43:53.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo update-grub 2023-12-16T21:43:54.029 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2023-12-16T21:43:54.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-16T21:43:54.097 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2023-12-16T21:43:54.264 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2023-12-16T21:43:54.265 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2023-12-16T21:43:54.409 INFO:teuthology.orchestra.run.smithi122.stdout:linux-image-generic is already the newest version (5.15.0.91.88). 2023-12-16T21:43:54.409 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:43:54.409 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-16T21:43:54.409 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-16T21:43:54.409 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-16T21:43:54.410 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-16T21:43:54.410 INFO:teuthology.orchestra.run.smithi122.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-16T21:43:54.410 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:43:54.448 INFO:teuthology.orchestra.run.smithi122.stdout:0 upgraded, 0 newly installed, 0 to remove and 277 not upgraded. 2023-12-16T21:43:54.450 DEBUG:teuthology.orchestra.run.smithi122:> dpkg -s linux-image-generic 2023-12-16T21:43:54.506 INFO:teuthology.orchestra.run.smithi122.stdout:Package: linux-image-generic 2023-12-16T21:43:54.506 INFO:teuthology.orchestra.run.smithi122.stdout:Status: install ok installed 2023-12-16T21:43:54.506 INFO:teuthology.orchestra.run.smithi122.stdout:Priority: optional 2023-12-16T21:43:54.506 INFO:teuthology.orchestra.run.smithi122.stdout:Section: kernel 2023-12-16T21:43:54.506 INFO:teuthology.orchestra.run.smithi122.stdout:Installed-Size: 20 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout:Maintainer: Ubuntu Kernel Team 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: amd64 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout:Source: linux-meta 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout:Version: 5.15.0.91.88 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout:Recommends: thermald 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout:Description: Generic Linux kernel image 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout: This package will always depend on the latest generic kernel image 2023-12-16T21:43:54.507 INFO:teuthology.orchestra.run.smithi122.stdout: available. 2023-12-16T21:43:54.507 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2023-12-16T21:43:54.551 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.zfNMiDkMCG 2023-12-16T21:43:54.552 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub/grub.cfg /tmp/tmp.zfNMiDkMCG 2023-12-16T21:43:54.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.zfNMiDkMCG 2023-12-16T21:43:54.733 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.zfNMiDkMCG is 10KB 2023-12-16T21:43:54.746 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.zfNMiDkMCG 2023-12-16T21:43:54.751 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-16T21:43:54.804 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:43:54.804 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-16T21:43:54.804 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-16T21:43:54.899 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-91-generic 2023-12-16T21:43:54.899 DEBUG:teuthology.orchestra.run.smithi122:> sudo update-grub 2023-12-16T21:43:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:Sourcing file `/etc/default/grub' 2023-12-16T21:43:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-16T21:43:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2023-12-16T21:43:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-16T21:43:55.399 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-16T21:43:55.629 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-16T21:43:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-16T21:43:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-16T21:43:55.685 INFO:teuthology.orchestra.run.smithi028.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-16T21:43:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-16T21:43:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-16T21:43:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-16T21:43:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:done 2023-12-16T21:43:55.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2023-12-16T21:43:56.179 INFO:teuthology.orchestra.run.smithi122.stderr:Sourcing file `/etc/default/grub' 2023-12-16T21:43:56.180 INFO:teuthology.orchestra.run.smithi122.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-16T21:43:56.185 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2023-12-16T21:43:56.381 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-16T21:43:56.386 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-16T21:43:56.675 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-16T21:43:56.681 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-16T21:43:56.722 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-16T21:43:56.728 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-16T21:43:56.869 INFO:teuthology.orchestra.run.smithi122.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-16T21:43:56.870 INFO:teuthology.orchestra.run.smithi122.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-16T21:43:56.870 INFO:teuthology.orchestra.run.smithi122.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-16T21:43:56.885 INFO:teuthology.orchestra.run.smithi122.stderr:done 2023-12-16T21:43:56.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2023-12-16T21:44:25.858 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-16T21:44:25.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2023-12-16T21:44:25.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:44:26.891 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-16T21:44:26.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-12-16T21:44:26.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:44:44.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2023-12-16T21:44:45.329 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-12-16T21:44:53.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2023-12-16T21:44:53.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:44:54.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-12-16T21:44:54.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:44:57.393 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-12-16T21:45:09.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-12-16T21:45:09.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:45:27.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-12-16T21:45:42.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-12-16T21:45:42.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:45:43.246 DEBUG:teuthology.orchestra.run.smithi122:> true 2023-12-16T21:45:43.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-12-16T21:45:43.828 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-91-generic"... 2023-12-16T21:45:43.828 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2023-12-16T21:45:43.873 INFO:teuthology.orchestra.run.smithi122.stdout:5.15.0-91-generic 2023-12-16T21:45:43.873 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-91-generic vs 5.15.0-91-generic 2023-12-16T21:45:43.873 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T21:45:43.873 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-16T21:45:44.874 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-16T21:45:44.875 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T21:45:44.951 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2023-12-16T21:45:44.983 DEBUG:teuthology.parallel:result is None 2023-12-16T21:45:53.318 DEBUG:teuthology.orchestra.remote:timed out 2023-12-16T21:46:05.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2023-12-16T21:46:05.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:46:05.593 DEBUG:teuthology.orchestra.run.smithi028:> true 2023-12-16T21:46:06.116 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2023-12-16T21:46:06.117 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-91-generic"... 2023-12-16T21:46:06.117 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2023-12-16T21:46:06.166 INFO:teuthology.orchestra.run.smithi028.stdout:5.15.0-91-generic 2023-12-16T21:46:06.166 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-91-generic vs 5.15.0-91-generic 2023-12-16T21:46:06.166 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T21:46:06.166 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-16T21:46:07.167 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-16T21:46:07.167 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T21:46:07.242 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2023-12-16T21:46:07.267 DEBUG:teuthology.parallel:result is None 2023-12-16T21:46:07.268 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-16T21:46:07.275 INFO:teuthology.task.internal:Creating test directory... 2023-12-16T21:46:07.275 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T21:46:07.278 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T21:46:07.284 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-16T21:46:07.292 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-16T21:46:07.303 INFO:teuthology.task.internal:Creating archive directory... 2023-12-16T21:46:07.303 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T21:46:07.329 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T21:46:07.419 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-16T21:46:07.427 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-16T21:46:07.427 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T21:46:07.430 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T21:46:07.447 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:46:07.448 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:46:07.455 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:46:07.456 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:46:07.457 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-16T21:46:07.466 INFO:teuthology.task.internal:Configuring sudo... 2023-12-16T21:46:07.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T21:46:07.503 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T21:46:07.518 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-16T21:46:07.531 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-16T21:46:07.532 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T21:46:07.563 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T21:46:07.569 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:46:07.615 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:46:07.662 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:46:07.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T21:46:07.724 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:46:07.729 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:46:07.777 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:46:07.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T21:46:07.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2023-12-16T21:46:07.853 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2023-12-16T21:46:07.931 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-16T21:46:07.941 INFO:teuthology.task.internal:Starting timer... 2023-12-16T21:46:07.944 INFO:teuthology.run_tasks:Running task pcp... 2023-12-16T21:46:07.958 INFO:teuthology.run_tasks:Running task selinux... 2023-12-16T21:46:07.967 DEBUG:teuthology.task.selinux:Excluding smithi028: OS 'ubuntu' does not support SELinux 2023-12-16T21:46:07.967 DEBUG:teuthology.task.selinux:Excluding smithi122: OS 'ubuntu' does not support SELinux 2023-12-16T21:46:07.967 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-16T21:46:07.967 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-16T21:46:07.967 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-16T21:46:07.967 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-16T21:46:07.979 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-16T21:46:07.984 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-16T21:46:08.101 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-16T21:46:08.117 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-16T21:46:08.118 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-16T21:53:58.921 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2023-12-16T21:53:58.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2023-12-16T21:53:58.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:53:59.005 DEBUG:teuthology.orchestra.run.smithi028:> true 2023-12-16T21:53:59.082 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2023-12-16T21:53:59.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-12-16T21:53:59.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:53:59.159 DEBUG:teuthology.orchestra.run.smithi122:> true 2023-12-16T21:53:59.237 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-12-16T21:53:59.238 INFO:teuthology.run_tasks:Running task clock... 2023-12-16T21:53:59.248 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-16T21:53:59.248 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T21:53:59.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:53:59.252 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T21:53:59.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:53:59.280 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-16T21:53:59.281 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Command line: ntpd -gq 2023-12-16T21:53:59.281 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: ---------------------------------------------------- 2023-12-16T21:53:59.281 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: ntp-4 is maintained by Network Time Foundation, 2023-12-16T21:53:59.281 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-16T21:53:59.281 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: corporation. Support and training for ntp-4 are 2023-12-16T21:53:59.281 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: available at https://www.nwtime.org/support 2023-12-16T21:53:59.281 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: ---------------------------------------------------- 2023-12-16T21:53:59.283 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: proto: precision = 0.067 usec (-24) 2023-12-16T21:53:59.283 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: basedate set to 2022-02-04 2023-12-16T21:53:59.283 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: gps base set to 2022-02-06 (week 2196) 2023-12-16T21:53:59.283 INFO:teuthology.orchestra.run.smithi028.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-16T21:53:59.283 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-16T21:53:59.284 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: restrict ::: KOD does nothing without LIMITED. 2023-12-16T21:53:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-16T21:53:59.285 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Listen and drop on 0 v6wildcard [::]:123 2023-12-16T21:53:59.285 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-16T21:53:59.285 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Listen normally on 2 lo 127.0.0.1:123 2023-12-16T21:53:59.285 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Listen normally on 3 ens1f0 172.21.15.28:123 2023-12-16T21:53:59.285 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Listen normally on 4 lo [::1]:123 2023-12-16T21:53:59.286 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:ca3c%4]:123 2023-12-16T21:53:59.286 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:53:59 ntpd[27897]: Listening on routing socket on fd #22 for interface updates 2023-12-16T21:53:59.307 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-16T21:53:59.308 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Command line: ntpd -gq 2023-12-16T21:53:59.308 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: ---------------------------------------------------- 2023-12-16T21:53:59.308 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: ntp-4 is maintained by Network Time Foundation, 2023-12-16T21:53:59.308 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-16T21:53:59.308 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: corporation. Support and training for ntp-4 are 2023-12-16T21:53:59.308 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: available at https://www.nwtime.org/support 2023-12-16T21:53:59.308 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: ---------------------------------------------------- 2023-12-16T21:53:59.309 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: proto: precision = 0.045 usec (-24) 2023-12-16T21:53:59.309 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: basedate set to 2022-02-04 2023-12-16T21:53:59.310 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: gps base set to 2022-02-06 (week 2196) 2023-12-16T21:53:59.310 INFO:teuthology.orchestra.run.smithi122.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-16T21:53:59.310 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-16T21:53:59.311 INFO:teuthology.orchestra.run.smithi122.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-16T21:53:59.311 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: restrict ::: KOD does nothing without LIMITED. 2023-12-16T21:53:59.312 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Listen and drop on 0 v6wildcard [::]:123 2023-12-16T21:53:59.312 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-16T21:53:59.312 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Listen normally on 2 lo 127.0.0.1:123 2023-12-16T21:53:59.312 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Listen normally on 3 enp3s0f1 172.21.15.122:123 2023-12-16T21:53:59.312 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Listen normally on 4 lo [::1]:123 2023-12-16T21:53:59.312 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d515%5]:123 2023-12-16T21:53:59.312 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:53:59 ntpd[28032]: Listening on routing socket on fd #22 for interface updates 2023-12-16T21:54:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:00 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:00 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:00 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:00 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:00 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:00 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:00 ntpd[27897]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:00 ntpd[27897]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:00.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:00.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:00.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:00.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:00.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:00.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:00.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:00.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:00.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:00.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:00.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:00.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:00.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:00.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:00 ntpd[28032]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:02 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:02 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:02 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:02 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:02 ntpd[27897]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:02 ntpd[27897]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:02.308 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:02.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:02 ntpd[28032]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:03.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:03 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:03.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:03 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:03.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:03 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:03.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:03 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:03.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:03 ntpd[27897]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:03 ntpd[27897]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:04.283 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.283 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.283 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:04.283 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:04 ntpd[27897]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:04.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:04.310 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:04 ntpd[28032]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:05 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:05 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:05 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:05 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:05 ntpd[27897]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:05 ntpd[27897]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:06.281 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stdout:16 Dec 21:54:06 ntpd[27897]: ntpd: time slew -0.001598 s 2023-12-16T21:54:06.282 INFO:teuthology.orchestra.run.smithi028.stdout:ntpd: time slew -0.001598s 2023-12-16T21:54:06.283 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-16T21:54:06.283 INFO:teuthology.orchestra.run.smithi028.stderr:16 Dec 21:54:06 ntpd[27897]: can't open /var/log/ntpstats/loopstats.20231216: Permission denied 2023-12-16T21:54:06.309 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:06.618 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:06.619 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:06.619 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:06.619 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:06.619 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:06.619 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T21:54:06.619 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T21:54:06.619 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T21:54:06.619 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T21:54:06.620 INFO:teuthology.orchestra.run.smithi122.stdout:16 Dec 21:54:06 ntpd[28032]: ntpd: time slew -0.000829 s 2023-12-16T21:54:06.620 INFO:teuthology.orchestra.run.smithi122.stdout:ntpd: time slew -0.000829s 2023-12-16T21:54:06.620 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-16T21:54:06.620 INFO:teuthology.orchestra.run.smithi122.stderr:16 Dec 21:54:06 ntpd[28032]: can't open /var/log/ntpstats/loopstats.20231216: Permission denied 2023-12-16T21:54:06.621 INFO:teuthology.orchestra.run.smithi122.stdout: remote refid st t when poll reach delay offset jitter 2023-12-16T21:54:06.621 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================== 2023-12-16T21:54:06.621 INFO:teuthology.orchestra.run.smithi122.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-16T21:54:06.622 INFO:teuthology.orchestra.run.smithi122.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-16T21:54:06.622 INFO:teuthology.orchestra.run.smithi122.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-16T21:54:06.622 INFO:teuthology.orchestra.run.smithi122.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-16T21:54:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: remote refid st t when poll reach delay offset jitter 2023-12-16T21:54:06.623 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================== 2023-12-16T21:54:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-16T21:54:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-16T21:54:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-16T21:54:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-16T21:54:06.623 INFO:teuthology.run_tasks:Running task install... 2023-12-16T21:54:06.648 DEBUG:teuthology.task.install:project ceph 2023-12-16T21:54:06.648 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}} 2023-12-16T21:54:06.648 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-16T21:54:06.649 INFO:teuthology.task.install:Using flavor: default 2023-12-16T21:54:06.655 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-16T21:54:06.656 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2023-12-16T21:54:06.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-key list | grep Ceph 2023-12-16T21:54:06.656 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-key list | grep Ceph 2023-12-16T21:54:06.724 INFO:teuthology.orchestra.run.smithi122.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-16T21:54:06.727 INFO:teuthology.orchestra.run.smithi028.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-16T21:54:06.754 INFO:teuthology.orchestra.run.smithi122.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-16T21:54:06.754 INFO:teuthology.orchestra.run.smithi122.stdout:uid [ unknown] Ceph.com (release key) 2023-12-16T21:54:06.754 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-16T21:54:06.754 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-12-16T21:54:06.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:54:06.759 INFO:teuthology.orchestra.run.smithi028.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-16T21:54:06.760 INFO:teuthology.orchestra.run.smithi028.stdout:uid [ unknown] Ceph.com (release key) 2023-12-16T21:54:06.760 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-16T21:54:06.760 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-12-16T21:54:06.760 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:54:06.975 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default/ 2023-12-16T21:54:06.976 INFO:teuthology.task.install.deb:Package version is 19.0.0-202-gc7586e95-1jammy 2023-12-16T21:54:06.986 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default/ 2023-12-16T21:54:06.986 INFO:teuthology.task.install.deb:Package version is 19.0.0-202-gc7586e95-1jammy 2023-12-16T21:54:07.098 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:54:07.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-16T21:54:07.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get update 2023-12-16T21:54:07.150 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:54:07.150 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-16T21:54:07.164 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2023-12-16T21:54:07.302 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-16T21:54:07.302 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-16T21:54:07.319 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-16T21:54:07.330 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-16T21:54:07.335 INFO:teuthology.orchestra.run.smithi028.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy InRelease 2023-12-16T21:54:07.338 INFO:teuthology.orchestra.run.smithi028.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-16T21:54:07.356 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-16T21:54:07.379 INFO:teuthology.orchestra.run.smithi028.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-16T21:54:07.381 INFO:teuthology.orchestra.run.smithi122.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy InRelease 2023-12-16T21:54:07.401 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-12-16T21:54:07.402 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-16T21:54:07.424 INFO:teuthology.orchestra.run.smithi122.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-16T21:54:07.429 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,390 B] 2023-12-16T21:54:07.445 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,390 B] 2023-12-16T21:54:07.451 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-16T21:54:07.472 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-12-16T21:54:07.531 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-16T21:54:07.613 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-16T21:54:07.671 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 28.5 kB in 0s (75.9 kB/s) 2023-12-16T21:54:07.750 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 28.5 kB in 0s (69.7 kB/s) 2023-12-16T21:54:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2023-12-16T21:54:08.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-202-gc7586e95-1jammy cephadm=19.0.0-202-gc7586e95-1jammy ceph-mds=19.0.0-202-gc7586e95-1jammy ceph-mgr=19.0.0-202-gc7586e95-1jammy ceph-common=19.0.0-202-gc7586e95-1jammy ceph-fuse=19.0.0-202-gc7586e95-1jammy ceph-test=19.0.0-202-gc7586e95-1jammy ceph-volume=19.0.0-202-gc7586e95-1jammy radosgw=19.0.0-202-gc7586e95-1jammy python3-rados=19.0.0-202-gc7586e95-1jammy python3-rgw=19.0.0-202-gc7586e95-1jammy python3-cephfs=19.0.0-202-gc7586e95-1jammy python3-rbd=19.0.0-202-gc7586e95-1jammy libcephfs2=19.0.0-202-gc7586e95-1jammy libcephfs-dev=19.0.0-202-gc7586e95-1jammy librados2=19.0.0-202-gc7586e95-1jammy librbd1=19.0.0-202-gc7586e95-1jammy rbd-fuse=19.0.0-202-gc7586e95-1jammy python3-cephfs=19.0.0-202-gc7586e95-1jammy cephfs-shell=19.0.0-202-gc7586e95-1jammy cephfs-top=19.0.0-202-gc7586e95-1jammy cephfs-mirror=19.0.0-202-gc7586e95-1jammy 2023-12-16T21:54:08.660 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2023-12-16T21:54:08.675 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2023-12-16T21:54:08.691 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-202-gc7586e95-1jammy cephadm=19.0.0-202-gc7586e95-1jammy ceph-mds=19.0.0-202-gc7586e95-1jammy ceph-mgr=19.0.0-202-gc7586e95-1jammy ceph-common=19.0.0-202-gc7586e95-1jammy ceph-fuse=19.0.0-202-gc7586e95-1jammy ceph-test=19.0.0-202-gc7586e95-1jammy ceph-volume=19.0.0-202-gc7586e95-1jammy radosgw=19.0.0-202-gc7586e95-1jammy python3-rados=19.0.0-202-gc7586e95-1jammy python3-rgw=19.0.0-202-gc7586e95-1jammy python3-cephfs=19.0.0-202-gc7586e95-1jammy python3-rbd=19.0.0-202-gc7586e95-1jammy libcephfs2=19.0.0-202-gc7586e95-1jammy libcephfs-dev=19.0.0-202-gc7586e95-1jammy librados2=19.0.0-202-gc7586e95-1jammy librbd1=19.0.0-202-gc7586e95-1jammy rbd-fuse=19.0.0-202-gc7586e95-1jammy python3-cephfs=19.0.0-202-gc7586e95-1jammy cephfs-shell=19.0.0-202-gc7586e95-1jammy cephfs-top=19.0.0-202-gc7586e95-1jammy cephfs-mirror=19.0.0-202-gc7586e95-1jammy 2023-12-16T21:54:08.751 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2023-12-16T21:54:08.833 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2023-12-16T21:54:08.833 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2023-12-16T21:54:08.947 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2023-12-16T21:54:08.947 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2023-12-16T21:54:08.989 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:54:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-16T21:54:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-16T21:54:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-16T21:54:08.989 INFO:teuthology.orchestra.run.smithi028.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-16T21:54:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-16T21:54:08.990 INFO:teuthology.orchestra.run.smithi028.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:54:08.990 INFO:teuthology.orchestra.run.smithi028.stdout:The following additional packages will be installed: 2023-12-16T21:54:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-16T21:54:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-16T21:54:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-16T21:54:08.990 INFO:teuthology.orchestra.run.smithi028.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-16T21:54:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-16T21:54:08.991 INFO:teuthology.orchestra.run.smithi028.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-12-16T21:54:08.992 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout:Suggested packages: 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-16T21:54:08.993 INFO:teuthology.orchestra.run.smithi028.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-16T21:54:08.994 INFO:teuthology.orchestra.run.smithi028.stdout:Recommended packages: 2023-12-16T21:54:08.994 INFO:teuthology.orchestra.run.smithi028.stdout: btrfs-tools 2023-12-16T21:54:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:The following NEW packages will be installed: 2023-12-16T21:54:09.101 INFO:teuthology.orchestra.run.smithi028.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-16T21:54:09.101 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-16T21:54:09.101 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-16T21:54:09.101 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-12-16T21:54:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-12-16T21:54:09.104 INFO:teuthology.orchestra.run.smithi028.stdout: socat unzip xmlstarlet zip 2023-12-16T21:54:09.126 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:54:09.126 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-16T21:54:09.126 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-16T21:54:09.127 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-16T21:54:09.127 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-16T21:54:09.127 INFO:teuthology.orchestra.run.smithi122.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-16T21:54:09.127 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:54:09.128 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2023-12-16T21:54:09.128 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-16T21:54:09.129 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-12-16T21:54:09.130 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-16T21:54:09.131 INFO:teuthology.orchestra.run.smithi122.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-16T21:54:09.132 INFO:teuthology.orchestra.run.smithi122.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-16T21:54:09.132 INFO:teuthology.orchestra.run.smithi122.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-16T21:54:09.132 INFO:teuthology.orchestra.run.smithi122.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-16T21:54:09.132 INFO:teuthology.orchestra.run.smithi122.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-16T21:54:09.132 INFO:teuthology.orchestra.run.smithi122.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-16T21:54:09.132 INFO:teuthology.orchestra.run.smithi122.stdout:Recommended packages: 2023-12-16T21:54:09.132 INFO:teuthology.orchestra.run.smithi122.stdout: btrfs-tools 2023-12-16T21:54:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:0 upgraded, 124 newly installed, 0 to remove and 265 not upgraded. 2023-12-16T21:54:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:Need to get 187 MB of archives. 2023-12-16T21:54:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:After this operation, 835 MB of additional disk space will be used. 2023-12-16T21:54:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-12-16T21:54:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-16T21:54:09.199 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-12-16T21:54:09.236 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2023-12-16T21:54:09.236 INFO:teuthology.orchestra.run.smithi122.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-16T21:54:09.236 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-16T21:54:09.236 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-16T21:54:09.240 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-16T21:54:09.241 INFO:teuthology.orchestra.run.smithi122.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-12-16T21:54:09.242 INFO:teuthology.orchestra.run.smithi122.stdout: socat unzip xmlstarlet zip 2023-12-16T21:54:09.247 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-16T21:54:09.249 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-202-gc7586e95-1jammy [3,460 kB] 2023-12-16T21:54:09.271 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-16T21:54:09.274 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-16T21:54:09.287 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-16T21:54:09.297 INFO:teuthology.orchestra.run.smithi122.stdout:0 upgraded, 124 newly installed, 0 to remove and 265 not upgraded. 2023-12-16T21:54:09.297 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 187 MB of archives. 2023-12-16T21:54:09.298 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 835 MB of additional disk space will be used. 2023-12-16T21:54:09.298 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-12-16T21:54:09.315 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-16T21:54:09.326 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-12-16T21:54:09.339 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-16T21:54:09.343 INFO:teuthology.orchestra.run.smithi028.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-16T21:54:09.355 INFO:teuthology.orchestra.run.smithi028.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-16T21:54:09.359 INFO:teuthology.orchestra.run.smithi028.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-16T21:54:09.359 INFO:teuthology.orchestra.run.smithi028.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-16T21:54:09.366 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-16T21:54:09.368 INFO:teuthology.orchestra.run.smithi028.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-16T21:54:09.370 INFO:teuthology.orchestra.run.smithi028.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-16T21:54:09.373 INFO:teuthology.orchestra.run.smithi028.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-16T21:54:09.374 INFO:teuthology.orchestra.run.smithi028.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-16T21:54:09.374 INFO:teuthology.orchestra.run.smithi028.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-16T21:54:09.381 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-202-gc7586e95-1jammy [3,460 kB] 2023-12-16T21:54:09.382 INFO:teuthology.orchestra.run.smithi028.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-16T21:54:09.385 INFO:teuthology.orchestra.run.smithi028.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-16T21:54:09.386 INFO:teuthology.orchestra.run.smithi028.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-16T21:54:09.388 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-16T21:54:09.390 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-16T21:54:09.402 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-16T21:54:09.403 INFO:teuthology.orchestra.run.smithi028.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-16T21:54:09.403 INFO:teuthology.orchestra.run.smithi028.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-16T21:54:09.404 INFO:teuthology.orchestra.run.smithi028.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-16T21:54:09.404 INFO:teuthology.orchestra.run.smithi028.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-16T21:54:09.404 INFO:teuthology.orchestra.run.smithi028.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-16T21:54:09.404 INFO:teuthology.orchestra.run.smithi028.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-16T21:54:09.405 INFO:teuthology.orchestra.run.smithi028.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-16T21:54:09.422 INFO:teuthology.orchestra.run.smithi028.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-16T21:54:09.422 INFO:teuthology.orchestra.run.smithi028.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-16T21:54:09.424 INFO:teuthology.orchestra.run.smithi028.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-16T21:54:09.446 INFO:teuthology.orchestra.run.smithi028.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-16T21:54:09.446 INFO:teuthology.orchestra.run.smithi028.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-16T21:54:09.446 INFO:teuthology.orchestra.run.smithi028.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-16T21:54:09.446 INFO:teuthology.orchestra.run.smithi028.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-16T21:54:09.446 INFO:teuthology.orchestra.run.smithi028.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-16T21:54:09.446 INFO:teuthology.orchestra.run.smithi028.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-16T21:54:09.446 INFO:teuthology.orchestra.run.smithi028.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-16T21:54:09.449 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-16T21:54:09.454 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-16T21:54:09.463 INFO:teuthology.orchestra.run.smithi028.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-16T21:54:09.463 INFO:teuthology.orchestra.run.smithi028.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-16T21:54:09.465 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-16T21:54:09.468 INFO:teuthology.orchestra.run.smithi028.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-16T21:54:09.469 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-16T21:54:09.469 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-16T21:54:09.478 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-16T21:54:09.480 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-16T21:54:09.482 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-16T21:54:09.482 INFO:teuthology.orchestra.run.smithi028.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-16T21:54:09.484 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-16T21:54:09.484 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-16T21:54:09.484 INFO:teuthology.orchestra.run.smithi028.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-16T21:54:09.484 INFO:teuthology.orchestra.run.smithi028.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-16T21:54:09.485 INFO:teuthology.orchestra.run.smithi028.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-16T21:54:09.487 INFO:teuthology.orchestra.run.smithi028.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-16T21:54:09.487 INFO:teuthology.orchestra.run.smithi028.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-16T21:54:09.489 INFO:teuthology.orchestra.run.smithi028.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-16T21:54:09.490 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-16T21:54:09.492 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-16T21:54:09.494 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-16T21:54:09.502 INFO:teuthology.orchestra.run.smithi028.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-16T21:54:09.504 INFO:teuthology.orchestra.run.smithi028.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-16T21:54:09.505 INFO:teuthology.orchestra.run.smithi028.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-16T21:54:09.508 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-16T21:54:09.509 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-16T21:54:09.510 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-16T21:54:09.510 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-16T21:54:09.510 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-16T21:54:09.511 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-16T21:54:09.511 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-16T21:54:09.522 INFO:teuthology.orchestra.run.smithi028.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-16T21:54:09.525 INFO:teuthology.orchestra.run.smithi028.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-16T21:54:09.526 INFO:teuthology.orchestra.run.smithi122.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-16T21:54:09.526 INFO:teuthology.orchestra.run.smithi122.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-16T21:54:09.529 INFO:teuthology.orchestra.run.smithi122.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-16T21:54:09.543 INFO:teuthology.orchestra.run.smithi122.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-16T21:54:09.543 INFO:teuthology.orchestra.run.smithi122.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-16T21:54:09.544 INFO:teuthology.orchestra.run.smithi122.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-16T21:54:09.545 INFO:teuthology.orchestra.run.smithi122.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-16T21:54:09.545 INFO:teuthology.orchestra.run.smithi122.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-16T21:54:09.545 INFO:teuthology.orchestra.run.smithi122.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-16T21:54:09.546 INFO:teuthology.orchestra.run.smithi122.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-16T21:54:09.547 INFO:teuthology.orchestra.run.smithi028.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-202-gc7586e95-1jammy [3,109 kB] 2023-12-16T21:54:09.561 INFO:teuthology.orchestra.run.smithi122.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-16T21:54:09.561 INFO:teuthology.orchestra.run.smithi122.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-16T21:54:09.566 INFO:teuthology.orchestra.run.smithi122.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-16T21:54:09.580 INFO:teuthology.orchestra.run.smithi122.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-16T21:54:09.581 INFO:teuthology.orchestra.run.smithi122.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-16T21:54:09.581 INFO:teuthology.orchestra.run.smithi122.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-16T21:54:09.582 INFO:teuthology.orchestra.run.smithi122.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-16T21:54:09.584 INFO:teuthology.orchestra.run.smithi122.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-16T21:54:09.585 INFO:teuthology.orchestra.run.smithi122.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-16T21:54:09.586 INFO:teuthology.orchestra.run.smithi028.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-16T21:54:09.586 INFO:teuthology.orchestra.run.smithi122.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-16T21:54:09.587 INFO:teuthology.orchestra.run.smithi028.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-16T21:54:09.589 INFO:teuthology.orchestra.run.smithi028.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-16T21:54:09.589 INFO:teuthology.orchestra.run.smithi028.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-16T21:54:09.591 INFO:teuthology.orchestra.run.smithi028.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-16T21:54:09.598 INFO:teuthology.orchestra.run.smithi122.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-16T21:54:09.600 INFO:teuthology.orchestra.run.smithi122.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-16T21:54:09.601 INFO:teuthology.orchestra.run.smithi122.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-16T21:54:09.613 INFO:teuthology.orchestra.run.smithi028.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-16T21:54:09.615 INFO:teuthology.orchestra.run.smithi028.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-16T21:54:09.615 INFO:teuthology.orchestra.run.smithi028.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-16T21:54:09.616 INFO:teuthology.orchestra.run.smithi122.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-16T21:54:09.618 INFO:teuthology.orchestra.run.smithi122.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-16T21:54:09.642 INFO:teuthology.orchestra.run.smithi028.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-16T21:54:09.642 INFO:teuthology.orchestra.run.smithi028.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-16T21:54:09.642 INFO:teuthology.orchestra.run.smithi028.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-16T21:54:09.643 INFO:teuthology.orchestra.run.smithi028.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-16T21:54:09.643 INFO:teuthology.orchestra.run.smithi028.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-16T21:54:09.644 INFO:teuthology.orchestra.run.smithi028.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-16T21:54:09.657 INFO:teuthology.orchestra.run.smithi028.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-16T21:54:09.658 INFO:teuthology.orchestra.run.smithi028.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-16T21:54:09.661 INFO:teuthology.orchestra.run.smithi028.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-16T21:54:09.668 INFO:teuthology.orchestra.run.smithi122.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-16T21:54:09.669 INFO:teuthology.orchestra.run.smithi122.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-16T21:54:09.671 INFO:teuthology.orchestra.run.smithi122.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-16T21:54:09.671 INFO:teuthology.orchestra.run.smithi122.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-16T21:54:09.672 INFO:teuthology.orchestra.run.smithi122.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-16T21:54:09.680 INFO:teuthology.orchestra.run.smithi028.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-16T21:54:09.681 INFO:teuthology.orchestra.run.smithi028.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-16T21:54:09.682 INFO:teuthology.orchestra.run.smithi028.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-16T21:54:09.685 INFO:teuthology.orchestra.run.smithi028.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-16T21:54:09.688 INFO:teuthology.orchestra.run.smithi028.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-12-16T21:54:09.688 INFO:teuthology.orchestra.run.smithi028.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-16T21:54:09.689 INFO:teuthology.orchestra.run.smithi122.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-16T21:54:09.689 INFO:teuthology.orchestra.run.smithi028.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-16T21:54:09.691 INFO:teuthology.orchestra.run.smithi122.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-16T21:54:09.691 INFO:teuthology.orchestra.run.smithi122.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-16T21:54:09.700 INFO:teuthology.orchestra.run.smithi028.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-16T21:54:09.703 INFO:teuthology.orchestra.run.smithi122.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-202-gc7586e95-1jammy [3,109 kB] 2023-12-16T21:54:09.705 INFO:teuthology.orchestra.run.smithi028.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-16T21:54:09.706 INFO:teuthology.orchestra.run.smithi028.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-16T21:54:09.706 INFO:teuthology.orchestra.run.smithi122.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-16T21:54:09.706 INFO:teuthology.orchestra.run.smithi122.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-16T21:54:09.711 INFO:teuthology.orchestra.run.smithi122.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-16T21:54:09.711 INFO:teuthology.orchestra.run.smithi122.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-16T21:54:09.712 INFO:teuthology.orchestra.run.smithi122.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-16T21:54:09.712 INFO:teuthology.orchestra.run.smithi122.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-16T21:54:09.713 INFO:teuthology.orchestra.run.smithi122.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-16T21:54:09.715 INFO:teuthology.orchestra.run.smithi028.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-202-gc7586e95-1jammy [690 kB] 2023-12-16T21:54:09.720 INFO:teuthology.orchestra.run.smithi028.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-16T21:54:09.721 INFO:teuthology.orchestra.run.smithi028.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-16T21:54:09.721 INFO:teuthology.orchestra.run.smithi028.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-16T21:54:09.721 INFO:teuthology.orchestra.run.smithi028.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-16T21:54:09.723 INFO:teuthology.orchestra.run.smithi028.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-16T21:54:09.724 INFO:teuthology.orchestra.run.smithi028.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-16T21:54:09.729 INFO:teuthology.orchestra.run.smithi122.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-16T21:54:09.731 INFO:teuthology.orchestra.run.smithi028.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-16T21:54:09.732 INFO:teuthology.orchestra.run.smithi122.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-16T21:54:09.734 INFO:teuthology.orchestra.run.smithi122.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-16T21:54:09.736 INFO:teuthology.orchestra.run.smithi122.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-16T21:54:09.736 INFO:teuthology.orchestra.run.smithi122.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-16T21:54:09.741 INFO:teuthology.orchestra.run.smithi028.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-16T21:54:09.741 INFO:teuthology.orchestra.run.smithi028.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-16T21:54:09.744 INFO:teuthology.orchestra.run.smithi028.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-16T21:54:09.746 INFO:teuthology.orchestra.run.smithi122.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-16T21:54:09.749 INFO:teuthology.orchestra.run.smithi122.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-12-16T21:54:09.749 INFO:teuthology.orchestra.run.smithi122.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-16T21:54:09.750 INFO:teuthology.orchestra.run.smithi122.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-16T21:54:09.751 INFO:teuthology.orchestra.run.smithi122.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-16T21:54:09.753 INFO:teuthology.orchestra.run.smithi028.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-202-gc7586e95-1jammy [356 kB] 2023-12-16T21:54:09.761 INFO:teuthology.orchestra.run.smithi028.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-16T21:54:09.763 INFO:teuthology.orchestra.run.smithi028.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-202-gc7586e95-1jammy [32.8 kB] 2023-12-16T21:54:09.763 INFO:teuthology.orchestra.run.smithi122.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-16T21:54:09.764 INFO:teuthology.orchestra.run.smithi122.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-16T21:54:09.764 INFO:teuthology.orchestra.run.smithi028.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-202-gc7586e95-1jammy [180 kB] 2023-12-16T21:54:09.767 INFO:teuthology.orchestra.run.smithi122.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-16T21:54:09.768 INFO:teuthology.orchestra.run.smithi122.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-16T21:54:09.768 INFO:teuthology.orchestra.run.smithi122.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-16T21:54:09.781 INFO:teuthology.orchestra.run.smithi122.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-16T21:54:09.783 INFO:teuthology.orchestra.run.smithi122.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-16T21:54:09.783 INFO:teuthology.orchestra.run.smithi122.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-16T21:54:09.788 INFO:teuthology.orchestra.run.smithi028.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-202-gc7586e95-1jammy [64.0 kB] 2023-12-16T21:54:09.790 INFO:teuthology.orchestra.run.smithi028.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-202-gc7586e95-1jammy [330 kB] 2023-12-16T21:54:09.791 INFO:teuthology.orchestra.run.smithi122.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-16T21:54:09.791 INFO:teuthology.orchestra.run.smithi122.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-16T21:54:09.798 INFO:teuthology.orchestra.run.smithi028.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-202-gc7586e95-1jammy [6,213 kB] 2023-12-16T21:54:09.799 INFO:teuthology.orchestra.run.smithi122.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-16T21:54:09.801 INFO:teuthology.orchestra.run.smithi122.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-16T21:54:09.802 INFO:teuthology.orchestra.run.smithi122.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-16T21:54:09.935 INFO:teuthology.orchestra.run.smithi122.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-202-gc7586e95-1jammy [690 kB] 2023-12-16T21:54:09.982 INFO:teuthology.orchestra.run.smithi122.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-202-gc7586e95-1jammy [356 kB] 2023-12-16T21:54:10.020 INFO:teuthology.orchestra.run.smithi122.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-202-gc7586e95-1jammy [32.8 kB] 2023-12-16T21:54:10.022 INFO:teuthology.orchestra.run.smithi122.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-202-gc7586e95-1jammy [180 kB] 2023-12-16T21:54:10.030 INFO:teuthology.orchestra.run.smithi122.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-202-gc7586e95-1jammy [64.0 kB] 2023-12-16T21:54:10.030 INFO:teuthology.orchestra.run.smithi122.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-202-gc7586e95-1jammy [330 kB] 2023-12-16T21:54:10.069 INFO:teuthology.orchestra.run.smithi122.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-202-gc7586e95-1jammy [6,213 kB] 2023-12-16T21:54:10.155 INFO:teuthology.orchestra.run.smithi028.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-202-gc7586e95-1jammy [112 kB] 2023-12-16T21:54:10.157 INFO:teuthology.orchestra.run.smithi028.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-202-gc7586e95-1jammy [439 kB] 2023-12-16T21:54:10.194 INFO:teuthology.orchestra.run.smithi028.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-202-gc7586e95-1jammy [22.7 MB] 2023-12-16T21:54:10.557 INFO:teuthology.orchestra.run.smithi122.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-202-gc7586e95-1jammy [112 kB] 2023-12-16T21:54:10.563 INFO:teuthology.orchestra.run.smithi122.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-202-gc7586e95-1jammy [439 kB] 2023-12-16T21:54:10.579 INFO:teuthology.orchestra.run.smithi122.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-202-gc7586e95-1jammy [22.7 MB] 2023-12-16T21:54:11.534 INFO:teuthology.orchestra.run.smithi028.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-202-gc7586e95-1jammy [6,899 kB] 2023-12-16T21:54:11.861 INFO:teuthology.orchestra.run.smithi028.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-202-gc7586e95-1jammy [238 kB] 2023-12-16T21:54:11.874 INFO:teuthology.orchestra.run.smithi028.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-202-gc7586e95-1jammy [124 kB] 2023-12-16T21:54:11.878 INFO:teuthology.orchestra.run.smithi028.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-202-gc7586e95-1jammy [1,493 kB] 2023-12-16T21:54:11.976 INFO:teuthology.orchestra.run.smithi028.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-202-gc7586e95-1jammy [9,959 kB] 2023-12-16T21:54:12.434 INFO:teuthology.orchestra.run.smithi122.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-202-gc7586e95-1jammy [6,899 kB] 2023-12-16T21:54:12.520 INFO:teuthology.orchestra.run.smithi028.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-202-gc7586e95-1jammy [31.9 MB] 2023-12-16T21:54:12.922 INFO:teuthology.orchestra.run.smithi122.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-202-gc7586e95-1jammy [238 kB] 2023-12-16T21:54:12.928 INFO:teuthology.orchestra.run.smithi122.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-202-gc7586e95-1jammy [124 kB] 2023-12-16T21:54:12.932 INFO:teuthology.orchestra.run.smithi122.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-202-gc7586e95-1jammy [1,493 kB] 2023-12-16T21:54:13.067 INFO:teuthology.orchestra.run.smithi122.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-202-gc7586e95-1jammy [9,959 kB] 2023-12-16T21:54:13.885 INFO:teuthology.orchestra.run.smithi122.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-202-gc7586e95-1jammy [31.9 MB] 2023-12-16T21:54:14.418 INFO:teuthology.orchestra.run.smithi028.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-202-gc7586e95-1jammy [14.1 kB] 2023-12-16T21:54:14.419 INFO:teuthology.orchestra.run.smithi028.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-202-gc7586e95-1jammy [839 kB] 2023-12-16T21:54:14.436 INFO:teuthology.orchestra.run.smithi028.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-202-gc7586e95-1jammy [2,145 kB] 2023-12-16T21:54:14.496 INFO:teuthology.orchestra.run.smithi028.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-202-gc7586e95-1jammy [833 kB] 2023-12-16T21:54:14.537 INFO:teuthology.orchestra.run.smithi028.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-202-gc7586e95-1jammy [139 kB] 2023-12-16T21:54:14.560 INFO:teuthology.orchestra.run.smithi028.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-202-gc7586e95-1jammy [1,905 kB] 2023-12-16T21:54:14.656 INFO:teuthology.orchestra.run.smithi028.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-202-gc7586e95-1jammy [8,625 kB] 2023-12-16T21:54:15.101 INFO:teuthology.orchestra.run.smithi028.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-202-gc7586e95-1jammy [14.2 kB] 2023-12-16T21:54:15.102 INFO:teuthology.orchestra.run.smithi028.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-202-gc7586e95-1jammy [48.6 MB] 2023-12-16T21:54:16.760 INFO:teuthology.orchestra.run.smithi122.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-202-gc7586e95-1jammy [14.1 kB] 2023-12-16T21:54:16.832 INFO:teuthology.orchestra.run.smithi122.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-202-gc7586e95-1jammy [839 kB] 2023-12-16T21:54:16.848 INFO:teuthology.orchestra.run.smithi122.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-202-gc7586e95-1jammy [2,145 kB] 2023-12-16T21:54:16.965 INFO:teuthology.orchestra.run.smithi122.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-202-gc7586e95-1jammy [833 kB] 2023-12-16T21:54:17.042 INFO:teuthology.orchestra.run.smithi122.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-202-gc7586e95-1jammy [139 kB] 2023-12-16T21:54:17.048 INFO:teuthology.orchestra.run.smithi122.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-202-gc7586e95-1jammy [1,905 kB] 2023-12-16T21:54:17.209 INFO:teuthology.orchestra.run.smithi122.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-202-gc7586e95-1jammy [8,625 kB] 2023-12-16T21:54:17.814 INFO:teuthology.orchestra.run.smithi028.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-202-gc7586e95-1jammy [121 kB] 2023-12-16T21:54:17.816 INFO:teuthology.orchestra.run.smithi028.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-202-gc7586e95-1jammy [225 kB] 2023-12-16T21:54:17.821 INFO:teuthology.orchestra.run.smithi028.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-202-gc7586e95-1jammy [34.0 kB] 2023-12-16T21:54:17.822 INFO:teuthology.orchestra.run.smithi028.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-202-gc7586e95-1jammy [23.4 kB] 2023-12-16T21:54:17.823 INFO:teuthology.orchestra.run.smithi028.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-202-gc7586e95-1jammy [38.3 kB] 2023-12-16T21:54:17.824 INFO:teuthology.orchestra.run.smithi028.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-202-gc7586e95-1jammy [12.3 MB] 2023-12-16T21:54:17.861 INFO:teuthology.orchestra.run.smithi122.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-202-gc7586e95-1jammy [14.2 kB] 2023-12-16T21:54:17.862 INFO:teuthology.orchestra.run.smithi122.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-202-gc7586e95-1jammy [48.6 MB] 2023-12-16T21:54:18.295 INFO:teuthology.orchestra.run.smithi028.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-202-gc7586e95-1jammy [91.4 kB] 2023-12-16T21:54:18.928 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 187 MB in 9s (20.4 MB/s) 2023-12-16T21:54:19.067 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-16T21:54:19.169 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2023-12-16T21:54:19.175 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-12-16T21:54:19.215 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-12-16T21:54:19.518 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-16T21:54:19.545 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-16T21:54:19.585 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-16T21:54:19.904 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cmd2. 2023-12-16T21:54:19.931 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-12-16T21:54:19.971 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-12-16T21:54:19.987 INFO:teuthology.orchestra.run.smithi122.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-202-gc7586e95-1jammy [121 kB] 2023-12-16T21:54:19.990 INFO:teuthology.orchestra.run.smithi122.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-202-gc7586e95-1jammy [225 kB] 2023-12-16T21:54:19.995 INFO:teuthology.orchestra.run.smithi122.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-202-gc7586e95-1jammy [34.0 kB] 2023-12-16T21:54:19.996 INFO:teuthology.orchestra.run.smithi122.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-202-gc7586e95-1jammy [23.4 kB] 2023-12-16T21:54:19.996 INFO:teuthology.orchestra.run.smithi122.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-202-gc7586e95-1jammy [38.3 kB] 2023-12-16T21:54:19.997 INFO:teuthology.orchestra.run.smithi122.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-202-gc7586e95-1jammy [12.3 MB] 2023-12-16T21:54:20.292 INFO:teuthology.orchestra.run.smithi122.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-202-gc7586e95-1jammy [91.4 kB] 2023-12-16T21:54:20.323 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-16T21:54:20.351 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-16T21:54:20.432 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-16T21:54:20.727 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-16T21:54:20.754 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-16T21:54:20.785 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-16T21:54:20.859 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 187 MB in 11s (16.9 MB/s) 2023-12-16T21:54:21.019 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-16T21:54:21.104 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-16T21:54:21.115 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2023-12-16T21:54:21.121 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-12-16T21:54:21.131 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-16T21:54:21.159 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-12-16T21:54:21.171 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-16T21:54:21.454 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-16T21:54:21.480 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-16T21:54:21.521 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-16T21:54:21.624 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-16T21:54:21.651 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-16T21:54:21.699 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:54:21.866 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cmd2. 2023-12-16T21:54:21.892 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-12-16T21:54:21.941 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-12-16T21:54:22.069 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-16T21:54:22.096 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-16T21:54:22.127 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:54:22.504 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-16T21:54:22.530 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-16T21:54:22.634 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-16T21:54:22.661 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-16T21:54:22.669 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-16T21:54:22.705 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:54:23.056 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-16T21:54:23.082 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-16T21:54:23.083 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-16T21:54:23.109 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-16T21:54:23.149 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-16T21:54:23.157 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-16T21:54:23.461 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librados2. 2023-12-16T21:54:23.477 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-16T21:54:23.487 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../010-librados2_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:23.504 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-16T21:54:23.527 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librados2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:23.543 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-16T21:54:23.921 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librbd1. 2023-12-16T21:54:23.937 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../011-librbd1_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:23.971 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librbd1 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:24.006 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-16T21:54:24.031 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-16T21:54:24.088 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:54:24.401 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcephfs2. 2023-12-16T21:54:24.426 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:24.465 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcephfs2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:24.484 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-16T21:54:24.510 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-16T21:54:24.551 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:54:24.860 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rados. 2023-12-16T21:54:24.886 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../013-python3-rados_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:24.896 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-16T21:54:24.922 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-16T21:54:24.926 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rados (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:24.954 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:54:25.262 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-16T21:54:25.289 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:25.316 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-16T21:54:25.328 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-ceph-argparse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:25.342 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-16T21:54:25.383 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-16T21:54:25.547 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cephfs. 2023-12-16T21:54:25.563 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:25.597 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cephfs (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:25.711 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librados2. 2023-12-16T21:54:25.737 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../010-librados2_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:25.777 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librados2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:25.917 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-16T21:54:25.943 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:25.983 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-ceph-common (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:26.189 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librbd1. 2023-12-16T21:54:26.204 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../011-librbd1_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:26.239 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librbd1 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:26.311 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-prettytable. 2023-12-16T21:54:26.338 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2023-12-16T21:54:26.378 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-16T21:54:26.647 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rbd. 2023-12-16T21:54:26.674 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:26.675 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libcephfs2. 2023-12-16T21:54:26.702 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:26.713 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rbd (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:26.742 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libcephfs2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:27.011 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rados. 2023-12-16T21:54:27.027 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../013-python3-rados_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:27.041 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-16T21:54:27.069 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-16T21:54:27.070 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rados (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:27.108 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-16T21:54:27.352 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-16T21:54:27.356 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-16T21:54:27.368 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-16T21:54:27.382 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:27.410 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-16T21:54:27.414 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-ceph-argparse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:27.651 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cephfs. 2023-12-16T21:54:27.677 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:27.852 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cephfs (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:28.105 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-16T21:54:28.132 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-16T21:54:28.171 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-16T21:54:28.172 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-16T21:54:28.197 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:28.238 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-ceph-common (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:28.483 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package lua5.1. 2023-12-16T21:54:28.514 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-16T21:54:28.549 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-16T21:54:28.600 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-prettytable. 2023-12-16T21:54:28.626 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2023-12-16T21:54:28.666 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-16T21:54:28.861 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package lua-any. 2023-12-16T21:54:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2023-12-16T21:54:28.902 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rbd. 2023-12-16T21:54:28.919 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-16T21:54:28.929 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:28.961 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rbd (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:29.113 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package zip. 2023-12-16T21:54:29.129 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2023-12-16T21:54:29.171 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking zip (3.0-12build2) ... 2023-12-16T21:54:29.314 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-16T21:54:29.340 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-16T21:54:29.381 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-16T21:54:29.491 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package unzip. 2023-12-16T21:54:29.518 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-16T21:54:29.557 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-16T21:54:29.675 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-16T21:54:29.690 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-16T21:54:29.733 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-16T21:54:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package luarocks. 2023-12-16T21:54:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-16T21:54:30.020 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-16T21:54:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-16T21:54:30.046 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-16T21:54:30.104 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-16T21:54:30.385 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package librgw2. 2023-12-16T21:54:30.412 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../027-librgw2_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:30.424 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package lua5.1. 2023-12-16T21:54:30.443 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking librgw2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:30.450 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-16T21:54:30.490 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-16T21:54:30.811 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package lua-any. 2023-12-16T21:54:30.826 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2023-12-16T21:54:30.869 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-16T21:54:30.954 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rgw. 2023-12-16T21:54:30.970 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:31.012 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rgw (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:31.105 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package zip. 2023-12-16T21:54:31.120 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2023-12-16T21:54:31.155 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking zip (3.0-12build2) ... 2023-12-16T21:54:31.415 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-16T21:54:31.442 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-16T21:54:31.459 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package unzip. 2023-12-16T21:54:31.485 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-16T21:54:31.490 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-16T21:54:31.526 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-16T21:54:31.859 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libradosstriper1. 2023-12-16T21:54:31.886 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:31.926 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libradosstriper1 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:31.938 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package luarocks. 2023-12-16T21:54:31.964 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-16T21:54:32.005 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-16T21:54:32.354 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-common. 2023-12-16T21:54:32.381 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../031-ceph-common_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-common (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:32.439 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librgw2. 2023-12-16T21:54:32.465 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../027-librgw2_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:32.513 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librgw2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:33.075 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rgw. 2023-12-16T21:54:33.101 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:33.141 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rgw (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:33.495 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-16T21:54:33.521 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-16T21:54:33.561 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-16T21:54:33.734 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-base. 2023-12-16T21:54:33.761 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../032-ceph-base_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:33.810 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-base (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:33.906 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libradosstriper1. 2023-12-16T21:54:33.932 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:33.981 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libradosstriper1 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:34.477 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-common. 2023-12-16T21:54:34.503 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../031-ceph-common_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:34.538 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-16T21:54:34.543 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-common (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:34.565 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-16T21:54:34.604 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-16T21:54:34.882 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cheroot. 2023-12-16T21:54:34.898 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-16T21:54:34.932 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-16T21:54:35.226 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-16T21:54:35.250 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-16T21:54:35.293 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-16T21:54:35.579 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-16T21:54:35.606 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-16T21:54:35.645 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-16T21:54:35.799 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-base. 2023-12-16T21:54:35.825 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../032-ceph-base_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:35.874 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-base (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:35.931 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-16T21:54:35.958 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-16T21:54:35.998 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-16T21:54:36.300 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-tempora. 2023-12-16T21:54:36.327 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2023-12-16T21:54:36.367 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-16T21:54:36.596 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-16T21:54:36.620 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-portend. 2023-12-16T21:54:36.622 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-16T21:54:36.647 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2023-12-16T21:54:36.662 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-16T21:54:36.678 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-16T21:54:36.966 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cheroot. 2023-12-16T21:54:37.073 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-16T21:54:37.073 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-16T21:54:37.073 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-16T21:54:37.073 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-16T21:54:37.073 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-16T21:54:37.361 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-16T21:54:37.387 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-16T21:54:37.416 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-16T21:54:37.427 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-16T21:54:37.444 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-16T21:54:37.492 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-16T21:54:37.689 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-16T21:54:37.715 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-16T21:54:37.755 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-16T21:54:37.862 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-natsort. 2023-12-16T21:54:37.889 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2023-12-16T21:54:37.928 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-16T21:54:38.234 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-16T21:54:38.260 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-16T21:54:38.281 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-logutils. 2023-12-16T21:54:38.301 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-16T21:54:38.308 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2023-12-16T21:54:38.356 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-16T21:54:38.587 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tempora. 2023-12-16T21:54:38.613 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2023-12-16T21:54:38.654 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-16T21:54:38.725 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-16T21:54:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-16T21:54:38.800 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-16T21:54:38.890 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-portend. 2023-12-16T21:54:38.905 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2023-12-16T21:54:38.949 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-16T21:54:39.145 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-mako. 2023-12-16T21:54:39.172 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-16T21:54:39.211 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-16T21:54:39.243 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-16T21:54:39.270 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-16T21:54:39.310 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-16T21:54:39.556 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-16T21:54:39.583 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-16T21:54:39.613 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-16T21:54:39.622 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-16T21:54:39.639 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-16T21:54:39.680 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-16T21:54:39.925 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-16T21:54:39.952 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-16T21:54:39.991 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-16T21:54:40.051 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-natsort. 2023-12-16T21:54:40.077 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2023-12-16T21:54:40.117 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-16T21:54:40.338 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-webob. 2023-12-16T21:54:40.365 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-16T21:54:40.413 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-16T21:54:40.429 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-logutils. 2023-12-16T21:54:40.455 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2023-12-16T21:54:40.495 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-16T21:54:40.774 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-16T21:54:40.791 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-waitress. 2023-12-16T21:54:40.800 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-16T21:54:40.818 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-16T21:54:40.832 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-16T21:54:40.860 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-16T21:54:41.119 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-mako. 2023-12-16T21:54:41.145 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-16T21:54:41.168 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-tempita. 2023-12-16T21:54:41.185 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-16T21:54:41.195 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-16T21:54:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-16T21:54:41.522 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-16T21:54:41.538 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-paste. 2023-12-16T21:54:41.548 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-16T21:54:41.565 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-16T21:54:41.589 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-16T21:54:41.604 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-16T21:54:41.892 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-16T21:54:41.918 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-16T21:54:41.924 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-16T21:54:41.940 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-16T21:54:41.975 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-16T21:54:41.982 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-16T21:54:42.220 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webob. 2023-12-16T21:54:42.246 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-16T21:54:42.372 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-16T21:54:42.477 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-16T21:54:42.504 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-16T21:54:42.844 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-16T21:54:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-waitress. 2023-12-16T21:54:43.350 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-16T21:54:43.394 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-16T21:54:43.397 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-webtest. 2023-12-16T21:54:43.424 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2023-12-16T21:54:43.463 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-16T21:54:43.669 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tempita. 2023-12-16T21:54:43.695 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-16T21:54:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-16T21:54:43.933 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pecan. 2023-12-16T21:54:43.960 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-16T21:54:43.991 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-16T21:54:44.013 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-paste. 2023-12-16T21:54:44.040 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-16T21:54:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-16T21:54:44.321 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-certifi. 2023-12-16T21:54:44.348 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2023-12-16T21:54:44.387 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-16T21:54:44.409 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-16T21:54:44.435 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-16T21:54:44.475 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-16T21:54:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-idna. 2023-12-16T21:54:44.720 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-16T21:54:44.733 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2023-12-16T21:54:44.735 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-16T21:54:44.770 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-16T21:54:44.773 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-16T21:54:45.032 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webtest. 2023-12-16T21:54:45.058 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2023-12-16T21:54:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-urllib3. 2023-12-16T21:54:45.098 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-16T21:54:45.111 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-16T21:54:45.151 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-16T21:54:45.418 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pecan. 2023-12-16T21:54:45.445 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-16T21:54:45.479 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-requests. 2023-12-16T21:54:45.485 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-16T21:54:45.506 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-16T21:54:45.545 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-16T21:54:45.832 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-16T21:54:45.840 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-certifi. 2023-12-16T21:54:45.859 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-16T21:54:45.867 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2023-12-16T21:54:45.898 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-16T21:54:45.906 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-16T21:54:46.210 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-16T21:54:46.218 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-idna. 2023-12-16T21:54:46.236 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:46.245 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2023-12-16T21:54:46.276 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-modules-core (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:46.285 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-16T21:54:46.572 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-urllib3. 2023-12-16T21:54:46.598 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-16T21:54:46.638 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-16T21:54:46.681 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-16T21:54:46.708 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:46.747 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:46.942 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-requests. 2023-12-16T21:54:46.968 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-16T21:54:47.008 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-16T21:54:47.066 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-16T21:54:47.093 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-16T21:54:47.133 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-16T21:54:47.328 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-16T21:54:47.356 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-16T21:54:47.395 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-16T21:54:47.435 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-openssl. 2023-12-16T21:54:47.463 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2023-12-16T21:54:47.502 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-16T21:54:47.757 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-16T21:54:47.772 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:47.806 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-modules-core (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:47.863 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr. 2023-12-16T21:54:47.891 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:47.930 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:48.187 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-16T21:54:48.214 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:48.254 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:48.266 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mon. 2023-12-16T21:54:48.293 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:48.324 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mon (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:48.724 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-16T21:54:48.750 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-16T21:54:48.790 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-16T21:54:49.051 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-openssl. 2023-12-16T21:54:49.067 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2023-12-16T21:54:49.110 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-16T21:54:49.203 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-osd. 2023-12-16T21:54:49.230 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:49.261 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-osd (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:49.455 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr. 2023-12-16T21:54:49.482 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:49.521 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:49.925 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mon. 2023-12-16T21:54:49.952 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:49.991 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mon (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:50.896 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-osd. 2023-12-16T21:54:50.922 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:50.962 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-osd (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:51.147 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph. 2023-12-16T21:54:51.174 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../068-ceph_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:51.214 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:51.516 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-fuse. 2023-12-16T21:54:51.543 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:51.582 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-fuse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:51.960 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mds. 2023-12-16T21:54:51.987 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:52.027 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mds (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:52.396 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephadm. 2023-12-16T21:54:52.424 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../071-cephadm_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:52.454 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephadm (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:52.707 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-16T21:54:52.723 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-16T21:54:52.757 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-16T21:54:52.782 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph. 2023-12-16T21:54:52.797 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../068-ceph_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:52.841 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:53.077 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-16T21:54:53.104 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-16T21:54:53.143 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-16T21:54:53.169 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-fuse. 2023-12-16T21:54:53.195 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:53.235 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-fuse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:53.580 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mds. 2023-12-16T21:54:53.607 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:53.647 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mds (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:54.067 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package cephadm. 2023-12-16T21:54:54.093 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../071-cephadm_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:54:54.133 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking cephadm (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:54.364 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-babel. 2023-12-16T21:54:54.392 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-16T21:54:54.414 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-16T21:54:54.431 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-16T21:54:54.440 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-16T21:54:54.472 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-16T21:54:54.784 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-jinja2. 2023-12-16T21:54:54.791 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-16T21:54:54.806 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-16T21:54:54.811 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2023-12-16T21:54:54.850 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-16T21:54:54.850 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-16T21:54:55.195 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-16T21:54:55.222 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:55.261 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-cephadm (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:55.615 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-16T21:54:55.642 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2023-12-16T21:54:55.681 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-16T21:54:55.906 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-babel. 2023-12-16T21:54:55.933 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-16T21:54:55.972 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-16T21:54:55.983 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-routes. 2023-12-16T21:54:56.011 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-16T21:54:56.050 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-16T21:54:56.300 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jinja2. 2023-12-16T21:54:56.328 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2023-12-16T21:54:56.367 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-16T21:54:56.395 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-16T21:54:56.422 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:56.461 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-dashboard (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:56.704 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-16T21:54:56.731 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:56.779 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-cephadm (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:56.890 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-16T21:54:56.917 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-16T21:54:56.956 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-16T21:54:57.116 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-16T21:54:57.143 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2023-12-16T21:54:57.182 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-16T21:54:57.468 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-joblib. 2023-12-16T21:54:57.485 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-routes. 2023-12-16T21:54:57.495 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-16T21:54:57.513 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-16T21:54:57.534 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-16T21:54:57.552 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-16T21:54:57.871 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-16T21:54:57.889 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-16T21:54:57.898 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-16T21:54:57.916 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:57.955 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-dashboard (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:58.023 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-16T21:54:58.325 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-sklearn. 2023-12-16T21:54:58.351 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-16T21:54:58.353 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-16T21:54:58.378 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-16T21:54:58.392 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-16T21:54:58.417 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-16T21:54:59.130 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-joblib. 2023-12-16T21:54:59.134 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-16T21:54:59.157 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-16T21:54:59.162 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:54:59.200 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:54:59.213 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-16T21:54:59.541 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-16T21:54:59.557 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-16T21:54:59.685 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-16T21:54:59.997 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sklearn. 2023-12-16T21:55:00.024 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-16T21:55:00.064 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-16T21:55:00.105 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-cachetools. 2023-12-16T21:55:00.131 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2023-12-16T21:55:00.169 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-16T21:55:00.498 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-16T21:55:00.525 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-16T21:55:00.564 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-16T21:55:00.716 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-16T21:55:00.744 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:00.774 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:00.884 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-16T21:55:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-16T21:55:00.950 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-16T21:55:01.270 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-rsa. 2023-12-16T21:55:01.297 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2023-12-16T21:55:01.336 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-16T21:55:01.690 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-google-auth. 2023-12-16T21:55:01.718 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2023-12-16T21:55:01.744 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cachetools. 2023-12-16T21:55:01.756 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-16T21:55:01.772 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2023-12-16T21:55:01.810 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-16T21:55:02.060 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-16T21:55:02.087 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-16T21:55:02.114 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-16T21:55:02.126 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-16T21:55:02.142 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-16T21:55:02.181 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-16T21:55:02.396 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-websocket. 2023-12-16T21:55:02.423 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2023-12-16T21:55:02.462 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-16T21:55:02.510 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-16T21:55:02.537 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-16T21:55:02.576 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-16T21:55:02.799 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-16T21:55:02.827 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-16T21:55:02.865 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-16T21:55:02.879 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rsa. 2023-12-16T21:55:02.906 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2023-12-16T21:55:02.954 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-16T21:55:03.275 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-google-auth. 2023-12-16T21:55:03.291 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2023-12-16T21:55:03.325 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-16T21:55:03.512 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-16T21:55:03.529 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:03.604 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-16T21:55:03.631 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-16T21:55:03.662 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:03.670 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-16T21:55:04.048 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-16T21:55:04.076 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-16T21:55:04.099 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-websocket. 2023-12-16T21:55:04.106 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-16T21:55:04.126 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2023-12-16T21:55:04.165 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-16T21:55:04.384 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-16T21:55:04.412 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-16T21:55:04.450 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-16T21:55:04.478 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-16T21:55:04.505 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-16T21:55:04.544 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-16T21:55:04.945 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package jq. 2023-12-16T21:55:04.973 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-16T21:55:05.011 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-16T21:55:05.250 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-16T21:55:05.266 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:05.289 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package socat. 2023-12-16T21:55:05.300 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:05.314 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-16T21:55:05.355 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-16T21:55:05.587 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-16T21:55:05.614 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-16T21:55:05.653 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-16T21:55:05.667 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package xmlstarlet. 2023-12-16T21:55:05.695 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-16T21:55:05.733 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-16T21:55:06.007 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-16T21:55:06.034 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-16T21:55:06.073 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-16T21:55:06.129 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-test. 2023-12-16T21:55:06.157 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../099-ceph-test_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:06.203 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-test (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:06.326 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package jq. 2023-12-16T21:55:06.353 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-16T21:55:06.393 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-16T21:55:06.604 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package socat. 2023-12-16T21:55:06.620 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-16T21:55:06.654 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-16T21:55:06.983 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package xmlstarlet. 2023-12-16T21:55:06.999 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-16T21:55:07.041 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-16T21:55:07.396 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-test. 2023-12-16T21:55:07.412 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../099-ceph-test_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:07.454 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-test (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:08.716 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ceph-volume. 2023-12-16T21:55:08.744 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:08.791 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ceph-volume (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:09.237 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-16T21:55:09.265 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:09.311 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephfs-mirror (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:09.631 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-colorama. 2023-12-16T21:55:09.659 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2023-12-16T21:55:09.697 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-12-16T21:55:09.775 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-volume. 2023-12-16T21:55:09.791 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:09.825 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-volume (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:09.958 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephfs-shell. 2023-12-16T21:55:09.975 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:10.016 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephfs-shell (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:10.205 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-16T21:55:10.232 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:10.271 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking cephfs-mirror (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:10.302 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package cephfs-top. 2023-12-16T21:55:10.330 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:10.377 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking cephfs-top (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:10.600 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-colorama. 2023-12-16T21:55:10.612 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-16T21:55:10.627 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2023-12-16T21:55:10.629 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:10.662 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcephfs-dev (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:10.666 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-12-16T21:55:10.952 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package cephfs-shell. 2023-12-16T21:55:10.968 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:10.999 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-16T21:55:11.010 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking cephfs-shell (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:11.027 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-16T21:55:11.065 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-16T21:55:11.247 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package cephfs-top. 2023-12-16T21:55:11.263 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-202-gc7586e95-1jammy_all.deb ... 2023-12-16T21:55:11.305 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking cephfs-top (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:11.428 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-16T21:55:11.456 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2023-12-16T21:55:11.494 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-16T21:55:11.550 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-16T21:55:11.566 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:11.608 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libcephfs-dev (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:11.856 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package nvme-cli. 2023-12-16T21:55:11.885 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-16T21:55:11.911 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-16T21:55:11.923 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-16T21:55:11.939 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-16T21:55:11.977 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-16T21:55:12.316 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-16T21:55:12.319 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package pkg-config. 2023-12-16T21:55:12.344 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2023-12-16T21:55:12.347 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-16T21:55:12.382 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-16T21:55:12.385 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-16T21:55:12.613 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-16T21:55:12.630 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-16T21:55:12.671 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-16T21:55:12.737 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package nvme-cli. 2023-12-16T21:55:12.765 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-16T21:55:12.804 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-16T21:55:13.058 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-16T21:55:13.086 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-16T21:55:13.116 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-16T21:55:13.218 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package pkg-config. 2023-12-16T21:55:13.246 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-16T21:55:13.284 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-16T21:55:13.402 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pastescript. 2023-12-16T21:55:13.430 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2023-12-16T21:55:13.468 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-16T21:55:13.546 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-16T21:55:13.562 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-16T21:55:13.604 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-16T21:55:13.797 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pluggy. 2023-12-16T21:55:13.825 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-16T21:55:13.863 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-16T21:55:14.041 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-16T21:55:14.069 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-16T21:55:14.108 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-16T21:55:14.166 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-psutil. 2023-12-16T21:55:14.195 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-16T21:55:14.232 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-16T21:55:14.386 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pastescript. 2023-12-16T21:55:14.402 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2023-12-16T21:55:14.435 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-16T21:55:14.577 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-py. 2023-12-16T21:55:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2023-12-16T21:55:14.653 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-16T21:55:14.748 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pluggy. 2023-12-16T21:55:14.776 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-16T21:55:14.806 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-16T21:55:14.972 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pygments. 2023-12-16T21:55:15.000 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-16T21:55:15.030 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-16T21:55:15.118 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-psutil. 2023-12-16T21:55:15.146 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-16T21:55:15.184 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-16T21:55:15.504 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-py. 2023-12-16T21:55:15.532 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2023-12-16T21:55:15.534 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-16T21:55:15.563 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-16T21:55:15.570 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-16T21:55:15.600 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-16T21:55:15.877 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pygments. 2023-12-16T21:55:15.903 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-16T21:55:15.941 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-16T21:55:15.944 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-toml. 2023-12-16T21:55:15.973 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2023-12-16T21:55:16.011 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-16T21:55:16.314 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-pytest. 2023-12-16T21:55:16.342 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-16T21:55:16.380 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-16T21:55:16.404 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-16T21:55:16.432 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-16T21:55:16.470 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-16T21:55:16.726 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package python3-simplejson. 2023-12-16T21:55:16.754 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-16T21:55:16.790 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-toml. 2023-12-16T21:55:16.791 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-16T21:55:16.818 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2023-12-16T21:55:16.857 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-16T21:55:17.061 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-16T21:55:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-16T21:55:17.119 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-16T21:55:17.135 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pytest. 2023-12-16T21:55:17.151 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-16T21:55:17.185 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-16T21:55:17.522 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-simplejson. 2023-12-16T21:55:17.550 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-16T21:55:17.588 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-16T21:55:17.806 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package radosgw. 2023-12-16T21:55:17.835 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../122-radosgw_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:17.864 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking radosgw (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:17.867 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-16T21:55:17.895 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-16T21:55:17.933 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-16T21:55:18.659 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package rbd-fuse. 2023-12-16T21:55:18.676 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:18.717 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking rbd-fuse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:18.754 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package radosgw. 2023-12-16T21:55:18.783 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../122-radosgw_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:18.821 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking radosgw (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:19.022 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-16T21:55:19.388 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-16T21:55:19.517 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-16T21:55:19.568 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package rbd-fuse. 2023-12-16T21:55:19.594 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-202-gc7586e95-1jammy_amd64.deb ... 2023-12-16T21:55:19.633 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking rbd-fuse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:19.778 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-16T21:55:19.938 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-16T21:55:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-16T21:55:20.396 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-16T21:55:20.531 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-16T21:55:20.548 INFO:teuthology.orchestra.run.smithi028.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-16T21:55:20.677 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephadm (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:20.764 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-16T21:55:21.076 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-16T21:55:21.312 INFO:teuthology.orchestra.run.smithi028.stdout:Adding system user cephadm....done 2023-12-16T21:55:21.458 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-16T21:55:21.510 INFO:teuthology.orchestra.run.smithi122.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-16T21:55:21.575 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up cephadm (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:21.735 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-16T21:55:22.019 INFO:teuthology.orchestra.run.smithi122.stdout:Adding system user cephadm....done 2023-12-16T21:55:22.027 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-16T21:55:22.153 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-16T21:55:22.174 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-16T21:55:22.446 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-16T21:55:22.475 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-16T21:55:22.706 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-16T21:55:22.726 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-16T21:55:22.832 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-16T21:55:22.853 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-16T21:55:23.104 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-16T21:55:23.158 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-16T21:55:23.363 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-16T21:55:23.482 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-16T21:55:23.527 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-16T21:55:23.758 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-16T21:55:23.770 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-12-16T21:55:24.030 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-16T21:55:24.119 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-16T21:55:24.289 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-16T21:55:24.387 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-12-16T21:55:24.541 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-16T21:55:24.647 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-16T21:55:24.809 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-16T21:55:24.923 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-16T21:55:25.068 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-16T21:55:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-16T21:55:25.311 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-ceph-argparse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:25.376 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-16T21:55:25.554 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-16T21:55:25.636 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-16T21:55:25.671 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-16T21:55:25.797 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-12-16T21:55:25.912 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-ceph-argparse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:26.056 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-16T21:55:26.172 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-16T21:55:26.298 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-16T21:55:26.307 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-16T21:55:26.408 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-12-16T21:55:26.567 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-16T21:55:26.650 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-16T21:55:26.776 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-16T21:55:26.928 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-16T21:55:26.957 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-16T21:55:27.027 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-16T21:55:27.028 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-16T21:55:27.154 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-16T21:55:27.396 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-16T21:55:27.426 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-16T21:55:27.489 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-16T21:55:27.506 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-16T21:55:27.591 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-16T21:55:27.632 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-16T21:55:27.916 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-16T21:55:27.925 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-16T21:55:28.052 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-16T21:55:28.184 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-16T21:55:28.311 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-16T21:55:28.510 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up zip (3.0-12build2) ... 2023-12-16T21:55:28.579 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-16T21:55:28.611 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-16T21:55:28.880 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up zip (3.0-12build2) ... 2023-12-16T21:55:28.999 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-16T21:55:29.321 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-16T21:55:29.581 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-16T21:55:29.626 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-16T21:55:29.690 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-16T21:55:29.817 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-16T21:55:29.877 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-16T21:55:30.028 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-16T21:55:30.117 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-16T21:55:30.154 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-16T21:55:30.447 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-16T21:55:30.510 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-16T21:55:30.753 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-16T21:55:30.849 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-16T21:55:31.092 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-16T21:55:31.121 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-16T21:55:31.414 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-12-16T21:55:31.460 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-16T21:55:31.765 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-16T21:55:31.778 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-12-16T21:55:32.079 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-16T21:55:32.125 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-16T21:55:32.384 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-16T21:55:32.423 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-16T21:55:32.510 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-ceph-common (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:32.674 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-16T21:55:32.800 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-ceph-common (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:32.812 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-idna (3.3-1) ... 2023-12-16T21:55:33.102 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-idna (3.3-1) ... 2023-12-16T21:55:33.138 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-16T21:55:33.436 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-16T21:55:34.344 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-16T21:55:34.570 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:55:34.618 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-16T21:55:34.705 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-16T21:55:34.870 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:55:34.972 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-16T21:55:35.004 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-16T21:55:35.098 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-16T21:55:35.233 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-16T21:55:35.264 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-16T21:55:35.543 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-16T21:55:35.623 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-16T21:55:35.743 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-16T21:55:35.802 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-16T21:55:36.036 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-16T21:55:36.044 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:55:36.181 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-16T21:55:36.295 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-16T21:55:36.497 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-16T21:55:36.546 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:55:36.655 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-16T21:55:36.756 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-16T21:55:36.965 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-16T21:55:37.033 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-16T21:55:37.200 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-16T21:55:37.309 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-16T21:55:37.459 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-16T21:55:37.568 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-16T21:55:37.710 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-16T21:55:37.836 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-16T21:55:37.961 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-16T21:55:37.962 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-16T21:55:38.229 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-16T21:55:38.238 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-16T21:55:38.355 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-16T21:55:38.364 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-16T21:55:38.623 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-16T21:55:38.657 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-16T21:55:38.724 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-16T21:55:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-16T21:55:38.991 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-16T21:55:39.234 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-16T21:55:39.301 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-16T21:55:39.369 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-16T21:55:39.470 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-16T21:55:39.542 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-16T21:55:39.552 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-16T21:55:39.679 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-16T21:55:39.804 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-16T21:55:39.804 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-16T21:55:39.893 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-16T21:55:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-16T21:55:40.130 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-16T21:55:40.240 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-16T21:55:40.290 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-16T21:55:40.550 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-16T21:55:40.600 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-16T21:55:40.676 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-16T21:55:40.885 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-16T21:55:40.935 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:55:41.010 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-16T21:55:41.036 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-16T21:55:41.270 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-16T21:55:41.346 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-16T21:55:41.396 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-16T21:55:41.663 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-16T21:55:41.748 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-16T21:55:41.948 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-16T21:55:42.048 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-16T21:55:42.176 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-16T21:55:42.299 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-16T21:55:42.486 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-16T21:55:42.494 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-16T21:55:42.754 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-16T21:55:42.761 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-16T21:55:43.013 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-16T21:55:43.029 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-16T21:55:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-16T21:55:43.399 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-16T21:55:43.649 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-16T21:55:43.700 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-16T21:55:43.934 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-16T21:55:44.069 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-16T21:55:44.170 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librados2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:44.278 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-16T21:55:44.296 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libsqlite3-mod-ceph (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:44.404 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librados2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:44.422 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-16T21:55:44.530 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libsqlite3-mod-ceph (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:44.648 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-16T21:55:44.784 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcephfs2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:44.910 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libradosstriper1 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:44.959 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libcephfs2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:45.036 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-16T21:55:45.102 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libradosstriper1 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:45.228 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-16T21:55:45.320 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librbd1 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:45.422 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-modules-core (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:45.529 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librbd1 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:45.539 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-fuse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:45.655 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-modules-core (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:45.760 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-16T21:55:45.761 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-16T21:55:45.798 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-fuse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:45.992 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-16T21:55:45.992 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-16T21:55:46.266 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcephfs-dev (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rados (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:46.466 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libcephfs-dev (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:46.592 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rados (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:46.718 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librgw2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:46.845 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up cephfs-top (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:46.869 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up librgw2 (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:46.970 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-16T21:55:46.995 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephfs-top (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:47.104 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-16T21:55:48.190 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rbd (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.291 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up rbd-fuse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.383 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rbd (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.426 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rgw (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.509 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up rbd-fuse (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.552 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cephfs (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-rgw (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.678 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up cephfs-shell (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.761 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up python3-cephfs (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.804 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-common (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:48.887 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephfs-shell (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:49.030 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-common (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:49.355 INFO:teuthology.orchestra.run.smithi122.stdout:Adding group ceph....done 2023-12-16T21:55:49.656 INFO:teuthology.orchestra.run.smithi028.stdout:Adding group ceph....done 2023-12-16T21:55:49.740 INFO:teuthology.orchestra.run.smithi122.stdout:Adding system user ceph....done 2023-12-16T21:55:49.867 INFO:teuthology.orchestra.run.smithi122.stdout:Setting system user ceph properties....done 2023-12-16T21:55:49.885 INFO:teuthology.orchestra.run.smithi122.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-16T21:55:50.028 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-16T21:55:50.041 INFO:teuthology.orchestra.run.smithi028.stdout:Adding system user ceph....done 2023-12-16T21:55:50.193 INFO:teuthology.orchestra.run.smithi028.stdout:Setting system user ceph properties....done 2023-12-16T21:55:50.210 INFO:teuthology.orchestra.run.smithi028.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-16T21:55:50.344 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-16T21:55:50.350 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-16T21:55:50.698 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-16T21:55:50.830 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up cephfs-mirror (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:51.225 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-16T21:55:51.225 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-16T21:55:51.307 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up cephfs-mirror (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:51.536 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-16T21:55:51.536 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-16T21:55:51.706 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-test (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:51.816 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up radosgw (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:52.033 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-test (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:52.161 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up radosgw (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:52.246 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:55:52.247 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:55:52.668 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:55:52.668 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:55:52.718 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-base (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:53.057 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-16T21:55:53.161 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-base (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:53.520 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-16T21:55:53.527 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mds (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:53.762 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-16T21:55:53.762 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-16T21:55:54.021 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mds (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:54.213 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:54.270 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-16T21:55:54.270 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-16T21:55:54.424 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-16T21:55:54.424 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-16T21:55:54.706 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:54.897 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-osd (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-16T21:55:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-16T21:55:55.172 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-16T21:55:55.172 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-16T21:55:55.433 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-osd (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:55.649 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-k8sevents (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:55.735 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-16T21:55:55.736 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-16T21:55:55.750 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:55.893 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mon (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:56.103 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-16T21:55:56.103 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-16T21:55:56.209 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-k8sevents (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:56.528 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:56.578 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-cephadm (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:56.721 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:56.729 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mon (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:56.863 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-dashboard (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:56.981 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-16T21:55:56.981 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-16T21:55:56.998 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-volume (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:57.325 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-16T21:55:57.435 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-16T21:55:57.473 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-cephadm (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:57.587 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-16T21:55:57.665 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:57.825 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-mgr-dashboard (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:57.976 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ceph-volume (19.0.0-202-gc7586e95-1jammy) ... 2023-12-16T21:55:58.329 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-16T21:55:58.421 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-16T21:55:58.565 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-16T21:55:58.846 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-16T21:55:59.487 INFO:teuthology.orchestra.run.smithi122.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-16T21:55:59.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-12-16T21:55:59.545 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2023-12-16T21:55:59.624 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-16T21:55:59.734 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2023-12-16T21:55:59.735 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2023-12-16T21:55:59.900 INFO:teuthology.orchestra.run.smithi122.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-12-16T21:55:59.900 INFO:teuthology.orchestra.run.smithi122.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-12-16T21:55:59.900 INFO:teuthology.orchestra.run.smithi122.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-12-16T21:55:59.900 INFO:teuthology.orchestra.run.smithi122.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-12-16T21:55:59.900 INFO:teuthology.orchestra.run.smithi122.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-12-16T21:55:59.901 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:55:59.901 INFO:teuthology.orchestra.run.smithi122.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-16T21:55:59.901 INFO:teuthology.orchestra.run.smithi122.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-16T21:55:59.901 INFO:teuthology.orchestra.run.smithi122.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-16T21:55:59.901 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-16T21:55:59.902 INFO:teuthology.orchestra.run.smithi122.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-16T21:55:59.902 INFO:teuthology.orchestra.run.smithi122.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:55:59.902 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2023-12-16T21:55:59.902 INFO:teuthology.orchestra.run.smithi122.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-12-16T21:55:59.902 INFO:teuthology.orchestra.run.smithi122.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-12-16T21:55:59.902 INFO:teuthology.orchestra.run.smithi122.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-12-16T21:55:59.902 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-12-16T21:55:59.902 INFO:teuthology.orchestra.run.smithi122.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-16T21:55:59.903 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2023-12-16T21:55:59.903 INFO:teuthology.orchestra.run.smithi122.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-12-16T21:55:59.903 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-12-16T21:55:59.903 INFO:teuthology.orchestra.run.smithi122.stdout: libengine-pkcs11-openssl 2023-12-16T21:55:59.940 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2023-12-16T21:55:59.941 INFO:teuthology.orchestra.run.smithi122.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-12-16T21:55:59.941 INFO:teuthology.orchestra.run.smithi122.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-12-16T21:55:59.941 INFO:teuthology.orchestra.run.smithi122.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-12-16T21:55:59.941 INFO:teuthology.orchestra.run.smithi122.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-12-16T21:55:59.941 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-12-16T21:55:59.941 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-12-16T21:55:59.941 INFO:teuthology.orchestra.run.smithi122.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-16T21:55:59.942 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages will be upgraded: 2023-12-16T21:55:59.942 INFO:teuthology.orchestra.run.smithi122.stdout: libssl3 2023-12-16T21:55:59.992 INFO:teuthology.orchestra.run.smithi122.stdout:1 upgraded, 39 newly installed, 0 to remove and 264 not upgraded. 2023-12-16T21:55:59.992 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 54.6 MB of archives. 2023-12-16T21:55:59.992 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 202 MB of additional disk space will be used. 2023-12-16T21:55:59.992 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-12-16T21:56:00.092 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-12-16T21:56:00.154 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-12-16T21:56:00.168 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-12-16T21:56:00.169 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-12-16T21:56:00.171 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-12-16T21:56:00.173 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-12-16T21:56:00.174 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2023-12-16T21:56:00.175 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-12-16T21:56:00.176 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-12-16T21:56:00.177 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-12-16T21:56:00.177 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-12-16T21:56:00.192 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-12-16T21:56:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-16T21:56:00.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-12-16T21:56:00.372 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2023-12-16T21:56:00.563 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-12-16T21:56:00.565 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-12-16T21:56:00.565 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-12-16T21:56:00.566 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2023-12-16T21:56:00.567 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2023-12-16T21:56:00.569 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-12-16T21:56:00.569 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-12-16T21:56:00.575 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.10-0ubuntu0.22.04.1 [143 kB] 2023-12-16T21:56:00.578 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-12-16T21:56:00.588 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-12-16T21:56:00.588 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-12-16T21:56:00.618 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-12-16T21:56:00.619 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-12-16T21:56:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-12-16T21:56:00.651 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-12-16T21:56:00.655 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-12-16T21:56:00.655 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-12-16T21:56:00.656 INFO:teuthology.orchestra.run.smithi122.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-12-16T21:56:00.656 INFO:teuthology.orchestra.run.smithi122.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.10-0ubuntu0.22.04.1 [1,223 kB] 2023-12-16T21:56:00.666 INFO:teuthology.orchestra.run.smithi122.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-12-16T21:56:00.668 INFO:teuthology.orchestra.run.smithi122.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.10-0ubuntu0.22.04.1 [16.1 MB] 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-16T21:56:00.743 INFO:teuthology.orchestra.run.smithi028.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-16T21:56:00.744 INFO:teuthology.orchestra.run.smithi028.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-16T21:56:00.744 INFO:teuthology.orchestra.run.smithi028.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-16T21:56:00.744 INFO:teuthology.orchestra.run.smithi028.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:56:00.745 INFO:teuthology.orchestra.run.smithi028.stdout:The following additional packages will be installed: 2023-12-16T21:56:00.745 INFO:teuthology.orchestra.run.smithi028.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-12-16T21:56:00.745 INFO:teuthology.orchestra.run.smithi028.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-12-16T21:56:00.746 INFO:teuthology.orchestra.run.smithi028.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-12-16T21:56:00.746 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-12-16T21:56:00.746 INFO:teuthology.orchestra.run.smithi028.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-16T21:56:00.747 INFO:teuthology.orchestra.run.smithi028.stdout:Suggested packages: 2023-12-16T21:56:00.747 INFO:teuthology.orchestra.run.smithi028.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-12-16T21:56:00.747 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-12-16T21:56:00.747 INFO:teuthology.orchestra.run.smithi028.stdout: libengine-pkcs11-openssl 2023-12-16T21:56:00.787 INFO:teuthology.orchestra.run.smithi028.stdout:The following NEW packages will be installed: 2023-12-16T21:56:00.787 INFO:teuthology.orchestra.run.smithi028.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-12-16T21:56:00.787 INFO:teuthology.orchestra.run.smithi028.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-12-16T21:56:00.788 INFO:teuthology.orchestra.run.smithi028.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-12-16T21:56:00.788 INFO:teuthology.orchestra.run.smithi028.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-12-16T21:56:00.788 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-12-16T21:56:00.788 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-12-16T21:56:00.788 INFO:teuthology.orchestra.run.smithi028.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-16T21:56:00.789 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages will be upgraded: 2023-12-16T21:56:00.790 INFO:teuthology.orchestra.run.smithi028.stdout: libssl3 2023-12-16T21:56:00.846 INFO:teuthology.orchestra.run.smithi028.stdout:1 upgraded, 39 newly installed, 0 to remove and 264 not upgraded. 2023-12-16T21:56:00.846 INFO:teuthology.orchestra.run.smithi028.stdout:Need to get 54.6 MB of archives. 2023-12-16T21:56:00.846 INFO:teuthology.orchestra.run.smithi028.stdout:After this operation, 202 MB of additional disk space will be used. 2023-12-16T21:56:00.847 INFO:teuthology.orchestra.run.smithi028.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-12-16T21:56:00.874 INFO:teuthology.orchestra.run.smithi122.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-12-16T21:56:00.875 INFO:teuthology.orchestra.run.smithi122.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-12-16T21:56:00.875 INFO:teuthology.orchestra.run.smithi122.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-12-16T21:56:00.875 INFO:teuthology.orchestra.run.smithi122.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-12-16T21:56:00.875 INFO:teuthology.orchestra.run.smithi122.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-12-16T21:56:00.875 INFO:teuthology.orchestra.run.smithi122.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-12-16T21:56:00.886 INFO:teuthology.orchestra.run.smithi122.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-12-16T21:56:00.887 INFO:teuthology.orchestra.run.smithi122.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-12-16T21:56:00.961 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-12-16T21:56:01.029 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-12-16T21:56:01.046 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-12-16T21:56:01.046 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-12-16T21:56:01.049 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-12-16T21:56:01.050 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-12-16T21:56:01.051 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2023-12-16T21:56:01.053 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-12-16T21:56:01.054 INFO:teuthology.orchestra.run.smithi028.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-12-16T21:56:01.055 INFO:teuthology.orchestra.run.smithi028.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-12-16T21:56:01.055 INFO:teuthology.orchestra.run.smithi028.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-12-16T21:56:01.072 INFO:teuthology.orchestra.run.smithi028.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-12-16T21:56:01.283 INFO:teuthology.orchestra.run.smithi122.stdout:Preconfiguring packages ... 2023-12-16T21:56:01.499 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 54.6 MB in 1s (58.2 MB/s) 2023-12-16T21:56:01.517 INFO:teuthology.orchestra.run.smithi028.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-12-16T21:56:01.519 INFO:teuthology.orchestra.run.smithi028.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-12-16T21:56:01.520 INFO:teuthology.orchestra.run.smithi028.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-12-16T21:56:01.524 INFO:teuthology.orchestra.run.smithi028.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-12-16T21:56:01.524 INFO:teuthology.orchestra.run.smithi028.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-12-16T21:56:01.530 INFO:teuthology.orchestra.run.smithi028.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.10-0ubuntu0.22.04.1 [143 kB] 2023-12-16T21:56:01.532 INFO:teuthology.orchestra.run.smithi028.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-12-16T21:56:01.543 INFO:teuthology.orchestra.run.smithi028.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-12-16T21:56:01.543 INFO:teuthology.orchestra.run.smithi028.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-12-16T21:56:01.573 INFO:teuthology.orchestra.run.smithi028.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-12-16T21:56:01.573 INFO:teuthology.orchestra.run.smithi028.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-12-16T21:56:01.585 INFO:teuthology.orchestra.run.smithi028.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-12-16T21:56:01.605 INFO:teuthology.orchestra.run.smithi028.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-12-16T21:56:01.612 INFO:teuthology.orchestra.run.smithi028.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-12-16T21:56:01.613 INFO:teuthology.orchestra.run.smithi028.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-12-16T21:56:01.614 INFO:teuthology.orchestra.run.smithi028.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-12-16T21:56:01.614 INFO:teuthology.orchestra.run.smithi028.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.10-0ubuntu0.22.04.1 [1,223 kB] 2023-12-16T21:56:01.632 INFO:teuthology.orchestra.run.smithi028.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-12-16T21:56:01.634 INFO:teuthology.orchestra.run.smithi028.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.10-0ubuntu0.22.04.1 [16.1 MB] 2023-12-16T21:56:01.655 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package flex. 2023-12-16T21:56:01.758 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173624 files and directories currently installed.) 2023-12-16T21:56:01.763 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-12-16T21:56:01.810 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking flex (2.6.4-8build2) ... 2023-12-16T21:56:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-12-16T21:56:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-12-16T21:56:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-12-16T21:56:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-12-16T21:56:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-12-16T21:56:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-12-16T21:56:01.966 INFO:teuthology.orchestra.run.smithi028.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-12-16T21:56:01.966 INFO:teuthology.orchestra.run.smithi028.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-12-16T21:56:02.388 INFO:teuthology.orchestra.run.smithi028.stdout:Preconfiguring packages ... 2023-12-16T21:56:02.401 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-16T21:56:02.552 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 54.6 MB in 1s (46.7 MB/s) 2023-12-16T21:56:02.574 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-12-16T21:56:02.673 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package flex. 2023-12-16T21:56:02.768 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173624 files and directories currently installed.) 2023-12-16T21:56:02.774 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-12-16T21:56:02.814 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking flex (2.6.4-8build2) ... 2023-12-16T21:56:02.977 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-16T21:56:03.347 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-16T21:56:03.509 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package bison. 2023-12-16T21:56:03.520 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-12-16T21:56:03.613 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173697 files and directories currently installed.) 2023-12-16T21:56:03.619 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-12-16T21:56:03.669 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-12-16T21:56:03.939 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-16T21:56:04.062 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-16T21:56:04.089 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-12-16T21:56:04.128 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-12-16T21:56:04.480 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package bison. 2023-12-16T21:56:04.498 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package dump. 2023-12-16T21:56:04.525 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-12-16T21:56:04.564 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking dump (0.4b47-3) ... 2023-12-16T21:56:04.583 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173697 files and directories currently installed.) 2023-12-16T21:56:04.589 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-12-16T21:56:04.632 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-12-16T21:56:04.818 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package indent. 2023-12-16T21:56:04.834 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:04.876 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-16T21:56:04.990 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-16T21:56:05.019 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-12-16T21:56:05.049 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-12-16T21:56:05.145 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-16T21:56:05.161 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-12-16T21:56:05.204 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-16T21:56:05.377 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package dump. 2023-12-16T21:56:05.405 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-12-16T21:56:05.443 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking dump (0.4b47-3) ... 2023-12-16T21:56:05.532 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-16T21:56:05.560 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2023-12-16T21:56:05.599 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2023-12-16T21:56:05.739 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package indent. 2023-12-16T21:56:05.755 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:05.797 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-16T21:56:05.860 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-16T21:56:05.876 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-12-16T21:56:05.910 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-12-16T21:56:06.066 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-16T21:56:06.083 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-12-16T21:56:06.124 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-16T21:56:06.221 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libfl2:amd64. 2023-12-16T21:56:06.249 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-12-16T21:56:06.288 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-12-16T21:56:06.395 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-16T21:56:06.423 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2023-12-16T21:56:06.453 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2023-12-16T21:56:06.523 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-12-16T21:56:06.540 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-12-16T21:56:06.582 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-16T21:56:06.714 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-16T21:56:06.731 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-12-16T21:56:06.772 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-12-16T21:56:06.928 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libjson-perl. 2023-12-16T21:56:06.955 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-12-16T21:56:06.985 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-12-16T21:56:07.202 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libfl2:amd64. 2023-12-16T21:56:07.231 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-12-16T21:56:07.263 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libllvm14:amd64. 2023-12-16T21:56:07.268 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-12-16T21:56:07.291 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-12-16T21:56:07.330 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-16T21:56:07.487 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-12-16T21:56:07.516 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-12-16T21:56:07.546 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-16T21:56:07.790 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libjson-perl. 2023-12-16T21:56:07.806 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-12-16T21:56:07.848 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-12-16T21:56:08.160 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libllvm14:amd64. 2023-12-16T21:56:08.188 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-12-16T21:56:08.226 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-16T21:56:08.983 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-16T21:56:09.011 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-16T21:56:09.049 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-16T21:56:09.319 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-16T21:56:09.346 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-16T21:56:09.385 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-16T21:56:09.679 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-16T21:56:09.707 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-16T21:56:09.738 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-16T21:56:09.871 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-16T21:56:09.899 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-16T21:56:09.937 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-16T21:56:10.116 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-16T21:56:10.143 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-12-16T21:56:10.173 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-16T21:56:10.182 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-16T21:56:10.202 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-16T21:56:10.240 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-16T21:56:10.510 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-16T21:56:10.537 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-12-16T21:56:10.576 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-16T21:56:10.584 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-16T21:56:10.613 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-16T21:56:10.659 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-16T21:56:10.929 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libpq5:amd64. 2023-12-16T21:56:10.957 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../16-libpq5_14.10-0ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:10.979 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-16T21:56:10.996 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:11.007 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-12-16T21:56:11.045 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-16T21:56:11.307 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-16T21:56:11.314 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-16T21:56:11.335 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:11.342 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-12-16T21:56:11.373 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-16T21:56:11.381 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-16T21:56:11.684 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-16T21:56:11.712 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:11.734 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libpq5:amd64. 2023-12-16T21:56:11.751 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-16T21:56:11.762 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../16-libpq5_14.10-0ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:11.800 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:12.012 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-16T21:56:12.028 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-16T21:56:12.070 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-16T21:56:12.105 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-16T21:56:12.133 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:12.205 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-16T21:56:12.533 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-16T21:56:12.549 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:12.591 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-16T21:56:12.607 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-16T21:56:12.635 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-12-16T21:56:12.674 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-12-16T21:56:12.818 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-16T21:56:12.835 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-16T21:56:12.877 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-16T21:56:12.976 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package modemmanager. 2023-12-16T21:56:13.004 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-16T21:56:13.043 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-16T21:56:13.397 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-16T21:56:13.426 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-12-16T21:56:13.465 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-12-16T21:56:13.683 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package network-manager. 2023-12-16T21:56:13.711 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-12-16T21:56:13.757 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-12-16T21:56:13.817 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package modemmanager. 2023-12-16T21:56:13.845 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-16T21:56:13.883 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-16T21:56:14.364 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ppp. 2023-12-16T21:56:14.392 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-12-16T21:56:14.498 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package network-manager. 2023-12-16T21:56:14.526 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-12-16T21:56:14.572 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-12-16T21:56:14.652 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-12-16T21:56:14.961 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package pptp-linux. 2023-12-16T21:56:14.988 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-12-16T21:56:15.027 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-12-16T21:56:15.197 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ppp. 2023-12-16T21:56:15.225 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-12-16T21:56:15.263 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-16T21:56:15.279 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-12-16T21:56:15.313 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-12-16T21:56:15.501 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-12-16T21:56:15.574 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package postgresql-client-common. 2023-12-16T21:56:15.602 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-12-16T21:56:15.632 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking postgresql-client-common (238) ... 2023-12-16T21:56:15.826 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package pptp-linux. 2023-12-16T21:56:15.855 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-12-16T21:56:15.893 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-12-16T21:56:15.945 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package postgresql-client-14. 2023-12-16T21:56:15.972 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../27-postgresql-client-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:16.028 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:16.154 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-16T21:56:16.183 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-12-16T21:56:16.220 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-12-16T21:56:16.538 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-client-common. 2023-12-16T21:56:16.567 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-12-16T21:56:16.607 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-client-common (238) ... 2023-12-16T21:56:16.607 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package postgresql-common. 2023-12-16T21:56:16.635 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-12-16T21:56:16.692 INFO:teuthology.orchestra.run.smithi122.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-12-16T21:56:16.713 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking postgresql-common (238) ... 2023-12-16T21:56:16.861 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-client-14. 2023-12-16T21:56:16.889 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../27-postgresql-client-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:16.927 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:17.175 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package postgresql-14. 2023-12-16T21:56:17.203 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../29-postgresql-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:17.244 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:17.558 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-common. 2023-12-16T21:56:17.587 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-12-16T21:56:17.652 INFO:teuthology.orchestra.run.smithi028.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-12-16T21:56:17.682 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-common (238) ... 2023-12-16T21:56:18.124 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-14. 2023-12-16T21:56:18.153 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../29-postgresql-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:56:18.193 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:18.611 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package postgresql. 2023-12-16T21:56:18.640 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-12-16T21:56:18.669 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking postgresql (14+238) ... 2023-12-16T21:56:18.879 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package postgresql-client. 2023-12-16T21:56:18.907 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-12-16T21:56:18.946 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking postgresql-client (14+238) ... 2023-12-16T21:56:19.193 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package postgresql-contrib. 2023-12-16T21:56:19.206 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-12-16T21:56:19.248 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking postgresql-contrib (14+238) ... 2023-12-16T21:56:19.551 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package tcl8.6. 2023-12-16T21:56:19.559 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql. 2023-12-16T21:56:19.579 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-12-16T21:56:19.587 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-12-16T21:56:19.618 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-16T21:56:19.618 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql (14+238) ... 2023-12-16T21:56:19.837 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-client. 2023-12-16T21:56:19.854 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-12-16T21:56:19.895 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-client (14+238) ... 2023-12-16T21:56:19.919 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package tcl. 2023-12-16T21:56:19.936 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-12-16T21:56:19.974 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-12-16T21:56:20.139 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package postgresql-contrib. 2023-12-16T21:56:20.156 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-12-16T21:56:20.197 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking postgresql-contrib (14+238) ... 2023-12-16T21:56:20.298 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package wpasupplicant. 2023-12-16T21:56:20.326 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-12-16T21:56:20.364 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-16T21:56:20.533 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package tcl8.6. 2023-12-16T21:56:20.562 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-12-16T21:56:20.624 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-16T21:56:20.735 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-16T21:56:20.751 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-12-16T21:56:20.793 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-12-16T21:56:20.886 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package tcl. 2023-12-16T21:56:20.902 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-12-16T21:56:20.941 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-12-16T21:56:21.138 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-16T21:56:21.166 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-12-16T21:56:21.213 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-16T21:56:21.280 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package wpasupplicant. 2023-12-16T21:56:21.309 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-12-16T21:56:21.347 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-16T21:56:21.485 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up postgresql-client-common (238) ... 2023-12-16T21:56:21.701 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-16T21:56:21.718 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-12-16T21:56:21.719 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up flex (2.6.4-8build2) ... 2023-12-16T21:56:21.751 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-12-16T21:56:21.847 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-16T21:56:21.990 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:22.138 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-16T21:56:22.166 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-12-16T21:56:22.166 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-12-16T21:56:22.212 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-16T21:56:22.784 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-16T21:56:22.827 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-client-common (238) ... 2023-12-16T21:56:22.902 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:23.170 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up flex (2.6.4-8build2) ... 2023-12-16T21:56:23.288 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-16T21:56:23.414 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:23.454 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-12-16T21:56:23.540 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-12-16T21:56:23.541 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-16T21:56:23.666 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-16T21:56:23.791 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:23.897 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-16T21:56:23.897 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-16T21:56:24.326 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-12-16T21:56:24.361 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-12-16T21:56:24.414 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-16T21:56:24.487 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-16T21:56:24.613 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-12-16T21:56:24.747 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-12-16T21:56:24.801 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-16T21:56:24.801 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-16T21:56:24.873 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-12-16T21:56:24.999 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-16T21:56:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-12-16T21:56:25.292 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-12-16T21:56:25.418 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-16T21:56:25.519 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-12-16T21:56:25.620 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-12-16T21:56:25.738 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-12-16T21:56:25.781 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-16T21:56:25.863 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-16T21:56:25.907 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up dump (0.4b47-3) ... 2023-12-16T21:56:25.990 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-12-16T21:56:26.000 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-16T21:56:26.067 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-16T21:56:26.176 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2023-12-16T21:56:26.294 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up tcl (8.6.11+1build2) ... 2023-12-16T21:56:26.420 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libjson-perl (4.04000-1) ... 2023-12-16T21:56:26.554 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-12-16T21:56:26.645 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-12-16T21:56:26.706 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-16T21:56:26.711 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-16T21:56:26.829 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up dump (0.4b47-3) ... 2023-12-16T21:56:26.831 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-16T21:56:26.923 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-16T21:56:26.989 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-16T21:56:26.999 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-16T21:56:27.101 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-16T21:56:27.115 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2023-12-16T21:56:27.218 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-12-16T21:56:27.241 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up tcl (8.6.11+1build2) ... 2023-12-16T21:56:27.367 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libjson-perl (4.04000-1) ... 2023-12-16T21:56:27.386 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up postgresql-client (14+238) ... 2023-12-16T21:56:27.493 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-12-16T21:56:27.537 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-12-16T21:56:27.574 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-12-16T21:56:27.628 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-16T21:56:27.737 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-16T21:56:27.914 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-16T21:56:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:28.026 INFO:teuthology.orchestra.run.smithi122.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-16T21:56:28.026 INFO:teuthology.orchestra.run.smithi122.stdout:which means they are currently configured by ifupdown: 2023-12-16T21:56:28.026 INFO:teuthology.orchestra.run.smithi122.stdout:- enp3s0f1 2023-12-16T21:56:28.026 INFO:teuthology.orchestra.run.smithi122.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-16T21:56:28.026 INFO:teuthology.orchestra.run.smithi122.stdout:remove their configuration from /etc/network/interfaces. 2023-12-16T21:56:28.026 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:28.056 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-16T21:56:28.141 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-16T21:56:28.182 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-12-16T21:56:28.350 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-client (14+238) ... 2023-12-16T21:56:28.439 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-16T21:56:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-12-16T21:56:28.756 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-16T21:56:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:28.926 INFO:teuthology.orchestra.run.smithi028.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-16T21:56:28.926 INFO:teuthology.orchestra.run.smithi028.stdout:which means they are currently configured by ifupdown: 2023-12-16T21:56:28.926 INFO:teuthology.orchestra.run.smithi028.stdout:- ens1f0 2023-12-16T21:56:28.926 INFO:teuthology.orchestra.run.smithi028.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-16T21:56:28.926 INFO:teuthology.orchestra.run.smithi028.stdout:remove their configuration from /etc/network/interfaces. 2023-12-16T21:56:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:29.059 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-16T21:56:29.224 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-16T21:56:29.375 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-16T21:56:29.385 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-16T21:56:29.476 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-12-16T21:56:29.602 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up postgresql-common (238) ... 2023-12-16T21:56:29.730 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-16T21:56:30.246 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-16T21:56:30.406 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-16T21:56:30.442 INFO:teuthology.orchestra.run.smithi122.stdout:Adding user postgres to group ssl-cert 2023-12-16T21:56:30.540 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-12-16T21:56:30.666 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:30.666 INFO:teuthology.orchestra.run.smithi122.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-12-16T21:56:30.667 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-common (238) ... 2023-12-16T21:56:30.864 INFO:teuthology.orchestra.run.smithi122.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-12-16T21:56:30.864 INFO:teuthology.orchestra.run.smithi122.stdout:Removing obsolete dictionary files: 2023-12-16T21:56:31.383 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-12-16T21:56:31.778 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:31.837 INFO:teuthology.orchestra.run.smithi028.stdout:Adding user postgres to group ssl-cert 2023-12-16T21:56:32.100 INFO:teuthology.orchestra.run.smithi122.stdout:Creating new PostgreSQL cluster 14/main ... 2023-12-16T21:56:32.118 INFO:teuthology.orchestra.run.smithi122.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout:The files belonging to this database system will be owned by user "postgres". 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout:This user must also own the server process. 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout:The database cluster will be initialized with locale "en_US". 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout:The default text search configuration will be set to "english". 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout:Data page checksums are disabled. 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:32.162 INFO:teuthology.orchestra.run.smithi122.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-12-16T21:56:32.163 INFO:teuthology.orchestra.run.smithi122.stdout:creating subdirectories ... ok 2023-12-16T21:56:32.163 INFO:teuthology.orchestra.run.smithi122.stdout:selecting dynamic shared memory implementation ... posix 2023-12-16T21:56:32.175 INFO:teuthology.orchestra.run.smithi122.stdout:selecting default max_connections ... 100 2023-12-16T21:56:32.196 INFO:teuthology.orchestra.run.smithi122.stdout:selecting default shared_buffers ... 128MB 2023-12-16T21:56:32.205 INFO:teuthology.orchestra.run.smithi122.stdout:selecting default time zone ... Etc/UTC 2023-12-16T21:56:32.206 INFO:teuthology.orchestra.run.smithi122.stdout:creating configuration files ... ok 2023-12-16T21:56:32.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:32.239 INFO:teuthology.orchestra.run.smithi028.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-12-16T21:56:32.321 INFO:teuthology.orchestra.run.smithi122.stdout:running bootstrap script ... ok 2023-12-16T21:56:32.598 INFO:teuthology.orchestra.run.smithi028.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-12-16T21:56:32.598 INFO:teuthology.orchestra.run.smithi028.stdout:Removing obsolete dictionary files: 2023-12-16T21:56:32.935 INFO:teuthology.orchestra.run.smithi122.stdout:performing post-bootstrap initialization ... ok 2023-12-16T21:56:33.177 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-12-16T21:56:33.623 INFO:teuthology.orchestra.run.smithi122.stdout:syncing data to disk ... ok 2023-12-16T21:56:33.886 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2023-12-16T21:56:34.003 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-12-16T21:56:34.307 INFO:teuthology.orchestra.run.smithi028.stdout:Creating new PostgreSQL cluster 14/main ... 2023-12-16T21:56:34.349 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-12-16T21:56:34.389 INFO:teuthology.orchestra.run.smithi028.stdout:The files belonging to this database system will be owned by user "postgres". 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:This user must also own the server process. 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:The database cluster will be initialized with locale "en_US". 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:The default text search configuration will be set to "english". 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:Data page checksums are disabled. 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:creating subdirectories ... ok 2023-12-16T21:56:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:selecting dynamic shared memory implementation ... posix 2023-12-16T21:56:34.407 INFO:teuthology.orchestra.run.smithi028.stdout:selecting default max_connections ... 100 2023-12-16T21:56:34.426 INFO:teuthology.orchestra.run.smithi028.stdout:selecting default shared_buffers ... 128MB 2023-12-16T21:56:34.433 INFO:teuthology.orchestra.run.smithi028.stdout:selecting default time zone ... Etc/UTC 2023-12-16T21:56:34.434 INFO:teuthology.orchestra.run.smithi028.stdout:creating configuration files ... ok 2023-12-16T21:56:34.545 INFO:teuthology.orchestra.run.smithi028.stdout:running bootstrap script ... ok 2023-12-16T21:56:35.183 INFO:teuthology.orchestra.run.smithi028.stdout:performing post-bootstrap initialization ... ok 2023-12-16T21:56:35.822 INFO:teuthology.orchestra.run.smithi028.stdout:syncing data to disk ... ok 2023-12-16T21:56:36.209 INFO:teuthology.orchestra.run.smithi028.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-12-16T21:56:36.744 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-16T21:56:36.853 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up postgresql-contrib (14+238) ... 2023-12-16T21:56:36.979 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up postgresql (14+238) ... 2023-12-16T21:56:37.114 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-16T21:56:37.240 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-16T21:56:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-16T21:56:37.497 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-16T21:56:38.076 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-16T21:56:38.395 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-16T21:56:38.538 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-16T21:56:39.010 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-16T21:56:39.136 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql-contrib (14+238) ... 2023-12-16T21:56:39.262 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up postgresql (14+238) ... 2023-12-16T21:56:39.388 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-16T21:56:39.489 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-16T21:56:39.732 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-16T21:56:39.732 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-16T21:56:40.316 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-16T21:56:40.375 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-12-16T21:56:40.483 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-16T21:56:40.619 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-16T21:56:40.770 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-16T21:56:40.995 INFO:teuthology.orchestra.run.smithi122.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-16T21:56:41.001 DEBUG:teuthology.parallel:result is None 2023-12-16T21:56:41.763 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-12-16T21:56:41.897 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-16T21:56:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-16T21:56:42.417 DEBUG:teuthology.parallel:result is None 2023-12-16T21:56:42.417 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:56:42.570 DEBUG:teuthology.orchestra.run.smithi028:> dpkg-query -W -f '${Version}' ceph 2023-12-16T21:56:42.590 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-202-gc7586e95-1jammy 2023-12-16T21:56:42.590 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-202-gc7586e95-1jammy 2023-12-16T21:56:42.590 INFO:teuthology.task.install:The correct ceph version 19.0.0-202-gc7586e95-1jammy is installed. 2023-12-16T21:56:42.592 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:56:42.811 DEBUG:teuthology.orchestra.run.smithi122:> dpkg-query -W -f '${Version}' ceph 2023-12-16T21:56:42.830 INFO:teuthology.orchestra.run.smithi122.stdout:19.0.0-202-gc7586e95-1jammy 2023-12-16T21:56:42.830 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-202-gc7586e95-1jammy 2023-12-16T21:56:42.831 INFO:teuthology.task.install:The correct ceph version 19.0.0-202-gc7586e95-1jammy is installed. 2023-12-16T21:56:42.832 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-16T21:56:42.832 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:42.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T21:56:42.847 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:56:42.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T21:56:42.888 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-16T21:56:42.888 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:42.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T21:56:42.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T21:56:42.958 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:56:42.958 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T21:56:42.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T21:56:43.026 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-16T21:56:43.026 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:43.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T21:56:43.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T21:56:43.094 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:56:43.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T21:56:43.107 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T21:56:43.162 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-16T21:56:43.163 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:43.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T21:56:43.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T21:56:43.230 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:56:43.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T21:56:43.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T21:56:43.299 INFO:teuthology.run_tasks:Running task ceph... 2023-12-16T21:56:43.366 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-16T21:56:43.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2023-12-16T21:56:43.368 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2023-12-16T21:56:43.383 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-16T21:56:43.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-16T21:56:43.427 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-16T21:56:43.440 INFO:tasks.ceph:Creating extra log directories... 2023-12-16T21:56:43.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-16T21:56:43.483 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-16T21:56:43.502 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-12-16T21:56:43.502 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-12-16T21:56:43.503 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-12-16T21:56:43.503 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:43.503 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-16T21:56:43.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-16T21:56:43.618 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:56:43.618 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-16T21:56:43.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-16T21:56:43.704 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-16T21:56:43.705 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-16T21:56:43.705 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888', 'branch': 'wip-yuri8-testing-2023-12-15-0911', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7494888', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri8-testing-2023-12-15-0911'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri8-testing-2023-12-15-0911', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2960, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-yuri8-testing-2023-12-15-0911', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHqnBEcUc0TyMy+JYxM1Vd8gvT1HQU6hovtByw5v0ToGSNJ0XNATQJQEQsRRZ3thjojEItyZw/kqsayZ20D4rSw=', 'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJ6URhJF7Vp3S0yxJdaNa9SgfnASuuzK9ws0P2U9Q88JHdZmDOM9uOsQdZ8pzetsGdReJKRBENC7VFvkslvzw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.tests_from_xfstests_dev']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-16_15:45:14', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-16T21:56:43.705 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-16T21:56:43.708 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-16T21:56:43.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2023-12-16T21:56:43.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2023-12-16T21:56:43.805 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:43.805 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T21:56:43.814 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T21:56:43.814 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2023-12-16T21:56:43.862 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T21:56:43.863 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:56:43.863 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 882 Links: 1 2023-12-16T21:56:43.863 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:56:43.863 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2023-12-16 21:50:38.436524636 +0000 2023-12-16T21:56:43.863 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2023-12-16 21:50:38.088527726 +0000 2023-12-16T21:56:43.863 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2023-12-16 21:50:38.088527726 +0000 2023-12-16T21:56:43.863 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: - 2023-12-16T21:56:43.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T21:56:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2023-12-16T21:56:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2023-12-16T21:56:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000257604 s, 2.0 MB/s 2023-12-16T21:56:43.919 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T21:56:43.968 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2023-12-16T21:56:44.014 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T21:56:44.015 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:56:44.015 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 906 Links: 1 2023-12-16T21:56:44.015 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:56:44.015 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2023-12-16 21:50:39.012519523 +0000 2023-12-16T21:56:44.015 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2023-12-16 21:50:38.656522683 +0000 2023-12-16T21:56:44.015 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2023-12-16 21:50:38.656522683 +0000 2023-12-16T21:56:44.015 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: - 2023-12-16T21:56:44.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T21:56:44.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2023-12-16T21:56:44.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2023-12-16T21:56:44.069 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000294091 s, 1.7 MB/s 2023-12-16T21:56:44.070 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T21:56:44.120 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2023-12-16T21:56:44.167 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T21:56:44.167 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:56:44.167 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 922 Links: 1 2023-12-16T21:56:44.167 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:56:44.167 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2023-12-16 21:50:39.596514338 +0000 2023-12-16T21:56:44.167 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2023-12-16 21:50:39.240517498 +0000 2023-12-16T21:56:44.167 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2023-12-16 21:50:39.240517498 +0000 2023-12-16T21:56:44.167 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: - 2023-12-16T21:56:44.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T21:56:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2023-12-16T21:56:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2023-12-16T21:56:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000288891 s, 1.8 MB/s 2023-12-16T21:56:44.222 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T21:56:44.272 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2023-12-16T21:56:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T21:56:44.318 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:56:44.318 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 938 Links: 1 2023-12-16T21:56:44.319 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:56:44.319 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2023-12-16 21:50:39.824512314 +0000 2023-12-16T21:56:44.319 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2023-12-16 21:50:39.820512349 +0000 2023-12-16T21:56:44.319 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2023-12-16 21:50:39.820512349 +0000 2023-12-16T21:56:44.319 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: - 2023-12-16T21:56:44.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T21:56:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2023-12-16T21:56:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2023-12-16T21:56:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.00020075 s, 2.6 MB/s 2023-12-16T21:56:44.374 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T21:56:44.424 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:56:44.424 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:56:44.424 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T21:56:44.430 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T21:56:44.430 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2023-12-16T21:56:44.475 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T21:56:44.476 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:56:44.476 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 877 Links: 1 2023-12-16T21:56:44.476 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:56:44.476 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-12-16 21:50:38.453432674 +0000 2023-12-16T21:56:44.476 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-12-16 21:50:38.101435817 +0000 2023-12-16T21:56:44.476 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-12-16 21:50:38.101435817 +0000 2023-12-16T21:56:44.476 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-12-16T21:56:44.476 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T21:56:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-12-16T21:56:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-12-16T21:56:44.530 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.00037455 s, 1.4 MB/s 2023-12-16T21:56:44.532 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T21:56:44.581 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2023-12-16T21:56:44.627 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T21:56:44.627 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:56:44.627 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-12-16T21:56:44.628 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:56:44.628 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-12-16 21:50:39.033427493 +0000 2023-12-16T21:56:44.628 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-12-16 21:50:38.693430529 +0000 2023-12-16T21:56:44.628 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-12-16 21:50:38.693430529 +0000 2023-12-16T21:56:44.628 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-12-16T21:56:44.628 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T21:56:44.681 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-12-16T21:56:44.681 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-12-16T21:56:44.681 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000304981 s, 1.7 MB/s 2023-12-16T21:56:44.682 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T21:56:44.729 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2023-12-16T21:56:44.775 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T21:56:44.775 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:56:44.775 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 913 Links: 1 2023-12-16T21:56:44.776 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:56:44.776 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-12-16 21:50:39.581422598 +0000 2023-12-16T21:56:44.776 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-12-16 21:50:39.245425598 +0000 2023-12-16T21:56:44.776 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-12-16 21:50:39.245425598 +0000 2023-12-16T21:56:44.776 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-12-16T21:56:44.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T21:56:44.830 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-12-16T21:56:44.830 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-12-16T21:56:44.830 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000407493 s, 1.3 MB/s 2023-12-16T21:56:44.832 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T21:56:44.881 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2023-12-16T21:56:44.928 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T21:56:44.928 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:56:44.928 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 933 Links: 1 2023-12-16T21:56:44.928 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:56:44.928 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-12-16 21:50:40.165417382 +0000 2023-12-16T21:56:44.928 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-12-16 21:50:39.829420383 +0000 2023-12-16T21:56:44.928 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-12-16 21:50:39.829420383 +0000 2023-12-16T21:56:44.928 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-12-16T21:56:44.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T21:56:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-12-16T21:56:44.981 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-12-16T21:56:44.982 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000368395 s, 1.4 MB/s 2023-12-16T21:56:44.983 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T21:56:45.033 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:56:45.033 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-16T21:56:45.033 INFO:tasks.ceph:Generating config... 2023-12-16T21:56:45.034 INFO:tasks.ceph:[client] client mount timeout = 600 2023-12-16T21:56:45.034 INFO:tasks.ceph:[client] debug client = 20 2023-12-16T21:56:45.035 INFO:tasks.ceph:[client] debug ms = 1 2023-12-16T21:56:45.035 INFO:tasks.ceph:[client] pid file = 2023-12-16T21:56:45.035 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-12-16T21:56:45.035 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] debug mds = 20 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] debug ms = 1 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] mds debug frag = True 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-16T21:56:45.035 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-16T21:56:45.036 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-16T21:56:45.036 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-16T21:56:45.036 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-16T21:56:45.036 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-16T21:56:45.036 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-16T21:56:45.037 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-16T21:56:45.037 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-16T21:56:45.037 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-16T21:56:45.037 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-12-16T21:56:45.037 INFO:tasks.ceph:Setting up mon.a... 2023-12-16T21:56:45.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-16T21:56:45.072 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2023-12-16T21:56:45.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-16T21:56:45.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-16T21:56:45.229 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28'), ('mon.b', '172.21.15.122'), ('mon.c', '[v2:172.21.15.122:3301,v1:172.21.15.122:6790]')] 2023-12-16T21:56:45.229 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28,172.21.15.122,[v2:172.21.15.122:3301,v1:172.21.15.122:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-12-16T21:56:45.230 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:45.230 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-16T21:56:45.275 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --add b 172.21.15.122 --addv c '[v2:172.21.15.122:3301,v1:172.21.15.122:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:56:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:56:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid fa65dfdd-b227-4a96-8047-b713453c357b 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = pacific 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:fsid fa65dfdd-b227-4a96-8047-b713453c357b 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2023-12-16T21:56:45.365130+0000 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:created 2023-12-16T21:56:45.365130+0000 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 16 (pacific) 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:2: [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] mon.c 2023-12-16T21:56:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-16T21:56:45.365 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-16T21:56:45.370 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fa65dfdd-b227-4a96-8047-b713453c357b... 2023-12-16T21:56:45.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-16T21:56:45.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-16T21:56:45.443 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2023-12-16T21:56:45.443 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-16T21:56:45.444 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2023-12-16T21:56:45.445 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2023-12-16T21:56:45.446 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.122,[v2:172.21.15.122:3301,v1:172.21.15.122:6790]" 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = fa65dfdd-b227-4a96-8047-b713453c357b 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2023-12-16T21:56:45.447 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug trim objects = true 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2023-12-16T21:56:45.448 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: bdev async discard = True 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: bdev enable discard = True 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore allocator = bitmap 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 1/20 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 1/20 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 4/10 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2023-12-16T21:56:45.449 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2023-12-16T21:56:45.450 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2023-12-16T21:56:45.451 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2023-12-16T21:56:45.452 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2023-12-16T21:56:45.453 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = True 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = True 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2023-12-16T21:56:45.454 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.455 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2023-12-16T21:56:45.456 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.457 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "172.21.15.28,172.21.15.122,[v2:172.21.15.122:3301,v1:172.21.15.122:6790]" 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = fa65dfdd-b227-4a96-8047-b713453c357b 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2023-12-16T21:56:45.458 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.459 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = bitmap 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1/20 2023-12-16T21:56:45.460 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1/20 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 4/10 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2023-12-16T21:56:45.461 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2023-12-16T21:56:45.462 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: mon op complaint time = 120 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2023-12-16T21:56:45.463 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: client mount timeout = 600 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: debug client = 20 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = "" 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout:[mds] 2023-12-16T21:56:45.464 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds = 20 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: debug mds balancer = 20 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug frag = True 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: mds debug scatterstat = True 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: mds op complaint time = 180 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: mds verify scatter = True 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: osd op complaint time = 180 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: rados mon op timeout = 900 2023-12-16T21:56:45.465 INFO:teuthology.orchestra.run.smithi122.stdout: rados osd op timeout = 900 2023-12-16T21:56:45.466 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-16T21:56:45.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-16T21:56:45.560 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-16T21:56:45.561 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:45.561 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-16T21:56:45.566 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:45.566 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-16T21:56:45.614 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2023-12-16T21:56:45.614 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:45.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-16T21:56:45.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-16T21:56:45.680 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:56:45.680 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:56:45.727 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2023-12-16T21:56:45.727 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:56:45.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-16T21:56:45.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-16T21:56:45.750 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:56:45.750 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:56:45.796 INFO:tasks.ceph:Setting up mon nodes... 2023-12-16T21:56:45.796 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-16T21:56:45.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-16T21:56:45.841 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-16T21:56:45.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-16T21:56:45.887 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-16T21:56:45.901 INFO:tasks.ceph:Setting up mds nodes... 2023-12-16T21:56:45.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-12-16T21:56:45.943 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-12-16T21:56:45.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-12-16T21:56:46.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-12-16T21:56:46.087 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-12-16T21:56:46.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-12-16T21:56:46.145 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-12-16T21:56:46.188 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-12-16T21:56:46.191 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-12-16T21:56:46.249 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-12-16T21:56:46.336 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-12-16T21:56:46.339 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-12-16T21:56:46.395 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-16T21:56:46.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-16T21:56:46.432 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-16T21:56:46.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-12-16T21:56:46.517 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-12-16T21:56:46.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-16T21:56:46.563 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-16T21:56:46.574 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-12-16T21:56:46.645 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-12-16T21:56:46.654 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-16T21:56:46.655 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-16T21:56:46.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-16T21:56:46.670 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:56:46.670 INFO:tasks.ceph:role: osd.0 2023-12-16T21:56:46.670 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2023-12-16T21:56:46.670 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-16T21:56:46.727 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:56:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:56:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:56:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-16T21:56:46.728 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:56:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:56:46.728 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:56:46.728 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-16T21:56:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:56:46.728 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:56:46.751 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2023-12-16T21:56:46.752 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2023-12-16T21:56:46.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-16T21:56:46.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-16T21:56:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:56:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2023-12-16T21:56:46.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-16T21:56:46.922 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:56:46.922 INFO:tasks.ceph:role: osd.1 2023-12-16T21:56:46.922 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2023-12-16T21:56:46.922 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-16T21:56:46.978 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:56:46.978 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:56:46.978 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:56:46.978 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-16T21:56:46.978 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:56:46.978 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:56:46.978 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:56:46.979 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-16T21:56:46.979 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:56:46.979 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:56:47.001 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2023-12-16T21:56:47.003 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2023-12-16T21:56:47.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-16T21:56:47.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-16T21:56:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:56:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2023-12-16T21:56:47.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-16T21:56:47.167 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:56:47.168 INFO:tasks.ceph:role: osd.2 2023-12-16T21:56:47.168 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com 2023-12-16T21:56:47.168 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-16T21:56:47.221 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:56:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:56:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:56:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-16T21:56:47.222 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:56:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:56:47.222 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:56:47.222 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-16T21:56:47.222 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:56:47.222 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:56:47.245 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2023-12-16T21:56:47.247 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2023-12-16T21:56:47.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-16T21:56:47.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-16T21:56:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:56:47.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2023-12-16T21:56:47.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-16T21:56:47.407 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:56:47.408 INFO:tasks.ceph:role: osd.3 2023-12-16T21:56:47.408 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com 2023-12-16T21:56:47.408 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-16T21:56:47.461 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:56:47.462 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:56:47.485 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2023-12-16T21:56:47.487 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2023-12-16T21:56:47.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-16T21:56:47.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-16T21:56:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:56:47.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: /sbin/restorecon: command not found 2023-12-16T21:56:47.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:56:47.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:47.710+0000 7f0ad63cc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-16T21:56:47.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:47.710+0000 7f0ad63cc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-16T21:56:47.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:47.710+0000 7f0ad63cc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-16T21:56:47.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:47.710+0000 7f0ad63cc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-16T21:56:47.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:47.710+0000 7f0ad63cc640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-16T21:56:47.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:47.710+0000 7f0ad63cc640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-16T21:56:50.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-16T21:56:50.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:56:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:50.822+0000 7fbb07115640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-16T21:56:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:50.826+0000 7fbb07115640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-16T21:56:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:50.826+0000 7fbb07115640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-16T21:56:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:50.826+0000 7fbb07115640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-16T21:56:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:50.826+0000 7fbb07115640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-16T21:56:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:50.826+0000 7fbb07115640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-16T21:56:53.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-16T21:56:53.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:56:53.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:53.970+0000 7f732df6b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-16T21:56:53.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:53.970+0000 7f732df6b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-16T21:56:53.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:53.970+0000 7f732df6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-16T21:56:53.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:53.970+0000 7f732df6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-16T21:56:53.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:53.970+0000 7f732df6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-16T21:56:53.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:53.970+0000 7f732df6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-16T21:56:56.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-16T21:56:57.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:56:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:57.122+0000 7f427a6df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-16T21:56:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:57.122+0000 7f427a6df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-16T21:56:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:57.122+0000 7f427a6df640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-16T21:56:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:57.122+0000 7f427a6df640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-16T21:56:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:57.122+0000 7f427a6df640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-16T21:56:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:56:57.122+0000 7f427a6df640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-16T21:57:00.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-16T21:57:00.175 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-16T21:57:00.190 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:57:00.190 INFO:tasks.ceph:role: osd.4 2023-12-16T21:57:00.190 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2023-12-16T21:57:00.190 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-16T21:57:00.250 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:57:00.250 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:57:00.250 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:57:00.250 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-16T21:57:00.251 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:57:00.251 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:57:00.251 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:57:00.251 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-16T21:57:00.251 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:57:00.251 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:57:00.718 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2023-12-16T21:57:00.720 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2023-12-16T21:57:00.721 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-16T21:57:00.828 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-16T21:57:00.839 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2023-12-16T21:57:00.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:57:00.840 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-16T21:57:00.893 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:57:00.893 INFO:tasks.ceph:role: osd.5 2023-12-16T21:57:00.893 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2023-12-16T21:57:00.893 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-16T21:57:00.955 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:57:00.956 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:57:01.382 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2023-12-16T21:57:01.385 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2023-12-16T21:57:01.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-16T21:57:01.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-16T21:57:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:57:01.501 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2023-12-16T21:57:01.501 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-16T21:57:01.556 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:57:01.556 INFO:tasks.ceph:role: osd.6 2023-12-16T21:57:01.556 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2023-12-16T21:57:01.557 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-16T21:57:01.619 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:57:01.620 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:57:02.083 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2023-12-16T21:57:02.085 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2023-12-16T21:57:02.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-16T21:57:02.172 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-16T21:57:02.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:57:02.225 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2023-12-16T21:57:02.226 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-16T21:57:02.278 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:57:02.278 INFO:tasks.ceph:role: osd.7 2023-12-16T21:57:02.278 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com 2023-12-16T21:57:02.278 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-16T21:57:02.339 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:57:02.340 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:57:02.795 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2023-12-16T21:57:02.797 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2023-12-16T21:57:02.797 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-16T21:57:02.914 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-16T21:57:02.925 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2023-12-16T21:57:02.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:57:02.926 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:57:03.029 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:03.025+0000 7f198fb56640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-16T21:57:03.033 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:03.029+0000 7f198fb56640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-16T21:57:03.033 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:03.029+0000 7f198fb56640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-16T21:57:03.033 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:03.029+0000 7f198fb56640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-16T21:57:03.033 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:03.029+0000 7f198fb56640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-16T21:57:03.039 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:03.037+0000 7f198fb56640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-16T21:57:06.080 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-16T21:57:06.093 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:57:06.173 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:06.169+0000 7f84ba3c9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-16T21:57:06.174 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:06.169+0000 7f84ba3c9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-16T21:57:06.174 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:06.169+0000 7f84ba3c9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-16T21:57:06.175 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:06.169+0000 7f84ba3c9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-16T21:57:06.175 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:06.169+0000 7f84ba3c9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-16T21:57:06.175 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:06.173+0000 7f84ba3c9640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-16T21:57:09.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-16T21:57:09.256 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:57:09.339 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:09.337+0000 7f24655d2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-16T21:57:09.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:09.337+0000 7f24655d2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-16T21:57:09.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:09.337+0000 7f24655d2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-16T21:57:09.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:09.337+0000 7f24655d2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-16T21:57:09.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:09.337+0000 7f24655d2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-16T21:57:09.340 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:09.337+0000 7f24655d2640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-16T21:57:12.352 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-16T21:57:12.412 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:57:12.494 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:12.489+0000 7f79133ff640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-16T21:57:12.495 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:12.493+0000 7f79133ff640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-16T21:57:12.495 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:12.493+0000 7f79133ff640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-16T21:57:12.495 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:12.493+0000 7f79133ff640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-16T21:57:12.495 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:12.493+0000 7f79133ff640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-16T21:57:12.495 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:12.493+0000 7f79133ff640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-16T21:57:13.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:57:13.709 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:57:15.523 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-16T21:57:15.537 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-16T21:57:15.537 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:15.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-16T21:57:15.552 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:15.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-12-16T21:57:15.605 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:15.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-12-16T21:57:15.662 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:15.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-16T21:57:15.717 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:15.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-16T21:57:15.773 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:15.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-16T21:57:15.829 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:15.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-16T21:57:15.886 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:15.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-16T21:57:15.900 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:15.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-12-16T21:57:15.954 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:15.954 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-12-16T21:57:16.010 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:16.011 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-16T21:57:16.067 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:16.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-16T21:57:16.122 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:16.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-16T21:57:16.179 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:16.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-16T21:57:16.234 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:16.234 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-16T21:57:16.240 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:16.240 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-12-16T21:57:16.286 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:16.286 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-16T21:57:16.292 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:16.292 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-12-16T21:57:16.339 INFO:tasks.ceph:Adding keys to all mons... 2023-12-16T21:57:16.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-16T21:57:16.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-16T21:57:16.385 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.y] 2023-12-16T21:57:16.385 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAdHX5l4hVQMhAA4+5NWEbViUXc1iOKVBLf/A== 2023-12-16T21:57:16.385 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2023-12-16T21:57:16.385 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAdHX5lNhVfOBAA/8ufBguEtvKEzSvyiAM76A== 2023-12-16T21:57:16.385 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2023-12-16T21:57:16.385 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAeHX5lKTRaBRAAeeuQC4/6XmX+rr4kF8hW6g== 2023-12-16T21:57:16.385 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2023-12-16T21:57:16.385 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAfHX5lk1STKhAACY4gGmaxvS+gtbtS/YaLvw== 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAiHX5lvCNjMRAAKFid+EnG5Kn3HL2c08AMcw== 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAlHX5lPycMOhAATI0Bl+hIinidrRZ+gavRVQ== 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQApHX5lx+iGBxAAJxGIuBGMpJlsm29neKGMFQ== 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAdHX5lCe/6NBAAgSGT10P74RECk7ODutoPtw== 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAeHX5lUYlUCxAAeDMP2IT7cmcsa4/6kRuw2A== 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAeHX5lOGooFBAAY3NgNlKKyMU91K87IMa6ig== 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.4] 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAvHX5l4fTYARAA5qFsU8D21FyFc4kIiRwJ7A== 2023-12-16T21:57:16.386 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.5] 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAyHX5l0CV7ChAA6LWW7LQC4ZVGrWRoVHCpzQ== 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.6] 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQA1HX5lnxBaFBAAl6/cEncJHyna8B1kWm0ZFA== 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.7] 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQA4HX5la9GcHRAAA720RxZp4IkNmVKnie1Ehg== 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAeHX5l+MjmGRAAJRtEVtccO6nez+DbuubEww== 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout:[client.2] 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAeHX5l7ZT+HhAAqOvNU4UQ8KmT7hw2/NfVww== 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout:[client.1] 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAeHX5lBSu0IRAAzzxW4dkqOsC0xhF4RfzG/g== 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout:[client.3] 2023-12-16T21:57:16.387 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAeHX5ltiOWJhAAtNM2X01MKH4/7p7QdHO/zg== 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.y] 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAdHX5l4hVQMhAA4+5NWEbViUXc1iOKVBLf/A== 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.a] 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAdHX5lNhVfOBAA/8ufBguEtvKEzSvyiAM76A== 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.c] 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAeHX5lKTRaBRAAeeuQC4/6XmX+rr4kF8hW6g== 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAfHX5lk1STKhAACY4gGmaxvS+gtbtS/YaLvw== 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAiHX5lvCNjMRAAKFid+EnG5Kn3HL2c08AMcw== 2023-12-16T21:57:16.393 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAlHX5lPycMOhAATI0Bl+hIinidrRZ+gavRVQ== 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.3] 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQApHX5lx+iGBxAAJxGIuBGMpJlsm29neKGMFQ== 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAdHX5lCe/6NBAAgSGT10P74RECk7ODutoPtw== 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.b] 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAeHX5lUYlUCxAAeDMP2IT7cmcsa4/6kRuw2A== 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout:[mds.d] 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAeHX5lOGooFBAAY3NgNlKKyMU91K87IMa6ig== 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.4] 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAvHX5l4fTYARAA5qFsU8D21FyFc4kIiRwJ7A== 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.5] 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAyHX5l0CV7ChAA6LWW7LQC4ZVGrWRoVHCpzQ== 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.6] 2023-12-16T21:57:16.394 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQA1HX5lnxBaFBAAl6/cEncJHyna8B1kWm0ZFA== 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.7] 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQA4HX5la9GcHRAAA720RxZp4IkNmVKnie1Ehg== 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAeHX5l+MjmGRAAJRtEVtccO6nez+DbuubEww== 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout:[client.2] 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAeHX5l7ZT+HhAAqOvNU4UQ8KmT7hw2/NfVww== 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAeHX5lBSu0IRAAzzxW4dkqOsC0xhF4RfzG/g== 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout:[client.3] 2023-12-16T21:57:16.395 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQAeHX5ltiOWJhAAtNM2X01MKH4/7p7QdHO/zg== 2023-12-16T21:57:16.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T21:57:16.433 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T21:57:16.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:57:16.519 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:57:16.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:57:16.607 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:57:16.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:16.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:16.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:16.740 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:16.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:16.827 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:16.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:16.907 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:16.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T21:57:16.989 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T21:57:17.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:57:17.076 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:57:17.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:57:17.171 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:57:17.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:17.242 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:17.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:17.326 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:17.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:17.412 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:17.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:17.498 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:57:17.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:57:17.582 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:57:17.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:57:17.671 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:57:17.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:57:17.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:57:17.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:57:17.808 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:57:17.889 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-16T21:57:17.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-16T21:57:17.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-16T21:57:18.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-16T21:57:18.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-16T21:57:18.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-16T21:57:18.125 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-16T21:57:18.135 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-16T21:57:18.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-16T21:57:18.281 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-16T21:57:18.291 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:57:18.294 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:57:18.339 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-16T21:57:18.339 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-16T21:57:18.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-16T21:57:18.343 INFO:tasks.ceph.mon.a:Started 2023-12-16T21:57:18.343 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-16T21:57:18.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-16T21:57:18.384 INFO:tasks.ceph.mon.b:Started 2023-12-16T21:57:18.384 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-16T21:57:18.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-16T21:57:18.386 INFO:tasks.ceph.mon.c:Started 2023-12-16T21:57:18.386 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-16T21:57:18.386 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-16T21:57:18.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-12-16T21:57:18.389 INFO:tasks.ceph.mgr.y:Started 2023-12-16T21:57:18.389 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-16T21:57:18.389 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-16T21:57:18.391 INFO:tasks.ceph.mgr.x:Started 2023-12-16T21:57:18.391 INFO:tasks.ceph:Setting crush tunables to default 2023-12-16T21:57:18.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2023-12-16T21:57:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb848b41640 1 Processor -- start 2023-12-16T21:57:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb848b41640 1 -- start start 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb848b41640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb8440cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb848b41640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb848b41640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8440d0570 con 0x7fb84415d2b0 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb848b41640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8440aff30 con 0x7fb8440d0120 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb848b41640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8440d1ec0 con 0x7fb84415d690 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb842d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb842d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40834/0 (socket says 172.21.15.28:40834) 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb842d9d640 1 -- 172.21.15.28:0/2160222047 learned_addr learned my addr 172.21.15.28:0/2160222047 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb8440cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb842d9d640 1 -- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 msgr2=0x7fb8440b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb842d9d640 1 -- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 msgr2=0x7fb8440b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb842d9d640 1 --2- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:57:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb842d9d640 1 --2- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:57:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb841d9b640 1 -- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 msgr2=0x7fb8440cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-16T21:57:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb841d9b640 1 -- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 msgr2=0x7fb8440cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:57:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb8440cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:57:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.514+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb8440cfbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:57:18.718 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb842d9d640 1 --2- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.718 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb8440cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.718 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb842d9d640 1 -- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 msgr2=0x7fb8440b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-16T21:57:18.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb842d9d640 1 -- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 msgr2=0x7fb8440b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:57:18.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb842d9d640 1 --2- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:57:18.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb842d9d640 1 --2- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:57:18.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb841d9b640 1 -- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 msgr2=0x7fb8440cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:57:18.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb841d9b640 1 -- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 msgr2=0x7fb8440cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:57:18.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb8440cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:57:18.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.718+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb8440cfbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:57:18.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.770+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171791520 0 0) 0x7fb834001120 con 0x7fb84415d2b0 2023-12-16T21:57:18.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.770+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c003530 con 0x7fb84415d2b0 2023-12-16T21:57:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1880251522 0 0) 0x7fb82c003530 con 0x7fb84415d2b0 2023-12-16T21:57:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 msgr2=0x7fb8440cfbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb8440cfbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 msgr2=0x7fb8440b0cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 --2- 172.21.15.28:0/2160222047 >> v2:172.21.15.28:3300/0 conn(0x7fb8440d0120 0x7fb8440b0cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8440b01d0 con 0x7fb84415d2b0 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fb834001810 con 0x7fb84415d2b0 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 >> v1:172.21.15.122:6789/0 conn(0x7fb84415d2b0 legacy=0x7fb8440d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb82c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c004000 0x7fb82c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb82c004d10 0x7fb82c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8440aff30 con 0x7fb82c004000 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8440d1ec0 con 0x7fb82c004d10 2023-12-16T21:57:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.774+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c00baf0 con 0x7fb84415d690 2023-12-16T21:57:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c004000 0x7fb82c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb84259c640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb82c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb842d9d640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb82c004d10 0x7fb82c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb841d9b640 1 -- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 msgr2=0x7fb82c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb82c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb841d9b640 1 -- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb82c004d10 msgr2=0x7fb82c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb82c004d10 0x7fb82c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb841d9b640 1 -- 172.21.15.28:0/2160222047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c003530 con 0x7fb82c004000 2023-12-16T21:57:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.778+0000 7fb84259c640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb84415d690 0x7fb82c003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb841d9b640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c004000 0x7fb82c0047d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb83800e8d0 tx=0x7fb83801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a5aecfaa5b6d95 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380a5750 con 0x7fb82c004000 2023-12-16T21:57:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8380a9070 con 0x7fb82c004000 2023-12-16T21:57:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb84159a640 1 -- 172.21.15.28:0/2160222047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380a5b30 con 0x7fb82c004000 2023-12-16T21:57:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb848b41640 1 -- 172.21.15.28:0/2160222047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c004000 msgr2=0x7fb82c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb848b41640 1 --2- 172.21.15.28:0/2160222047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c004000 0x7fb82c0047d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb83800e8d0 tx=0x7fb83801c850 comp rx=0 tx=0).stop 2023-12-16T21:57:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb842d9d640 1 -- 172.21.15.28:0/2160222047 reap_dead start 2023-12-16T21:57:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb848b41640 1 -- 172.21.15.28:0/2160222047 shutdown_connections 2023-12-16T21:57:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb848b41640 1 -- 172.21.15.28:0/2160222047 >> 172.21.15.28:0/2160222047 conn(0x7fb84409c510 msgr2=0x7fb8440d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb848b41640 1 -- 172.21.15.28:0/2160222047 shutdown_connections 2023-12-16T21:57:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.782+0000 7fb848b41640 1 -- 172.21.15.28:0/2160222047 wait complete. 2023-12-16T21:57:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb848b41640 1 Processor -- start 2023-12-16T21:57:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb848b41640 1 -- start start 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb848b41640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb82c004000 0x7fb84415aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb848b41640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8440b75b0 0x7fb8440b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb848b41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8441f2e00 0x7fb8441f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb848b41640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8440b7f70 con 0x7fb8441f2e00 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb848b41640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84415b030 con 0x7fb8440b75b0 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb848b41640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8440b6e90 con 0x7fb82c004000 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb842d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8441f2e00 0x7fb8441f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb841d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8440b75b0 0x7fb8440b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb842d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8441f2e00 0x7fb8441f3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40876/0 (socket says 172.21.15.28:40876) 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb842d9d640 1 -- 172.21.15.28:0/649865665 learned_addr learned my addr 172.21.15.28:0/649865665 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.786+0000 7fb84259c640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb82c004000 0x7fb84415aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb842d9d640 1 -- 172.21.15.28:0/649865665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb82c004000 msgr2=0x7fb84415aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb842d9d640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb82c004000 0x7fb84415aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb842d9d640 1 -- 172.21.15.28:0/649865665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8440b75b0 msgr2=0x7fb8440b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb842d9d640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8440b75b0 0x7fb8440b7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb842d9d640 1 -- 172.21.15.28:0/649865665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8440b5df0 con 0x7fb8441f2e00 2023-12-16T21:57:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb841d9b640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8440b75b0 0x7fb8440b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb842d9d640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8441f2e00 0x7fb8441f3280 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb84408ce90 tx=0x7fb83c007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb832ffd640 1 -- 172.21.15.28:0/649865665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c010bc0 con 0x7fb8441f2e00 2023-12-16T21:57:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb832ffd640 1 -- 172.21.15.28:0/649865665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83c00f040 con 0x7fb8441f2e00 2023-12-16T21:57:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb832ffd640 1 -- 172.21.15.28:0/649865665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c015440 con 0x7fb8441f2e00 2023-12-16T21:57:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8440b60d0 con 0x7fb8441f2e00 2023-12-16T21:57:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8440b6690 con 0x7fb8441f2e00 2023-12-16T21:57:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb832ffd640 1 -- 172.21.15.28:0/649865665 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb83c0158a0 con 0x7fb8441f2e00 2023-12-16T21:57:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb832ffd640 1 -- 172.21.15.28:0/649865665 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb83c01e910 con 0x7fb8441f2e00 2023-12-16T21:57:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.790+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8440679f0 con 0x7fb8441f2e00 2023-12-16T21:57:18.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.794+0000 7fb832ffd640 1 -- 172.21.15.28:0/649865665 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7fb83c01e410 con 0x7fb8441f2e00 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.858+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb8440cefa0 con 0x7fb8441f2e00 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb832ffd640 1 -- 172.21.15.28:0/649865665 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb83c015ce0 con 0x7fb8441f2e00 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8441f2e00 msgr2=0x7fb8441f3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8441f2e00 0x7fb8441f3280 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb84408ce90 tx=0x7fb83c007e20 comp rx=0 tx=0).stop 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 shutdown_connections 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8441f2e00 0x7fb8441f3280 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8440b75b0 0x7fb8440b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 --2- 172.21.15.28:0/649865665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb82c004000 0x7fb84415aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 >> 172.21.15.28:0/649865665 conn(0x7fb84409c510 msgr2=0x7fb8440b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 shutdown_connections 2023-12-16T21:57:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:18.862+0000 7fb848b41640 1 -- 172.21.15.28:0/649865665 wait complete. 2023-12-16T21:57:18.882 INFO:tasks.ceph:check_enable_crimson: False 2023-12-16T21:57:18.882 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-16T21:57:18.882 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:18.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-16T21:57:18.893 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:18.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-16T21:57:18.947 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:18.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-16T21:57:18.978 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:18.973+0000 7fca5fa10180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T21:57:18.979 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:18.978+0000 7ff19134b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T21:57:18.988 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:57:18.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-16T21:57:19.039 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:19.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-16T21:57:19.052 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:19.052 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-16T21:57:19.105 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:19.105 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-16T21:57:19.157 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:57:19.157 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-16T21:57:19.210 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 814fdfdf-9ed9-440c-814d-7d73b1a21072 0 2023-12-16T21:57:19.373 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb38106640 1 Processor -- start 2023-12-16T21:57:19.373 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb38106640 1 -- start start 2023-12-16T21:57:19.374 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb38106640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb3006de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb38106640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb3006bc50 0x7feb300ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb38106640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3015a4f0 con 0x7feb300aeb80 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb38106640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb30154960 con 0x7feb3006bc50 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb38106640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3006e6c0 con 0x7feb30154580 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb356a1640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7feb300aeb80 0x7feb3006b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48674/0 (socket says 172.21.15.122:48674) 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb35ea2640 1 --2- 172.21.15.122:0/494171801 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb3006de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.369+0000 7feb356a1640 1 -- 172.21.15.122:0/494171801 learned_addr learned my addr 172.21.15.122:0/494171801 (peer_addr_for_me v1:172.21.15.122:0/0) 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb366a3640 1 --2- 172.21.15.122:0/494171801 >> v2:172.21.15.28:3300/0 conn(0x7feb3006bc50 0x7feb300ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb34ea0640 1 -- 172.21.15.122:0/494171801 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858227713 0 0) 0x7feb3015a4f0 con 0x7feb300aeb80 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb34ea0640 1 -- 172.21.15.122:0/494171801 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb1c003530 con 0x7feb300aeb80 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb35ea2640 1 -- 172.21.15.122:0/494171801 >> v2:172.21.15.28:3300/0 conn(0x7feb3006bc50 msgr2=0x7feb300ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb35ea2640 1 --2- 172.21.15.122:0/494171801 >> v2:172.21.15.28:3300/0 conn(0x7feb3006bc50 0x7feb300ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb35ea2640 1 -- 172.21.15.122:0/494171801 >> v1:172.21.15.122:6789/0 conn(0x7feb300aeb80 legacy=0x7feb3006b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.375 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb35ea2640 1 -- 172.21.15.122:0/494171801 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3015a740 con 0x7feb30154580 2023-12-16T21:57:19.377 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb35ea2640 1 --2- 172.21.15.122:0/494171801 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb3006de90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0097c0 tx=0x7feb2c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c169bc6f0409d75 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:19.377 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb34ea0640 1 -- 172.21.15.122:0/494171801 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0ac620 con 0x7feb30154580 2023-12-16T21:57:19.377 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb34ea0640 1 -- 172.21.15.122:0/494171801 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c0ac780 con 0x7feb30154580 2023-12-16T21:57:19.377 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb34ea0640 1 -- 172.21.15.122:0/494171801 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c0acb10 con 0x7feb30154580 2023-12-16T21:57:19.378 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb38106640 1 -- 172.21.15.122:0/494171801 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 msgr2=0x7feb3006de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.378 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb38106640 1 --2- 172.21.15.122:0/494171801 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb3006de90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0097c0 tx=0x7feb2c01c3c0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.378 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb38106640 1 -- 172.21.15.122:0/494171801 shutdown_connections 2023-12-16T21:57:19.378 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb38106640 1 --2- 172.21.15.122:0/494171801 >> v2:172.21.15.28:3300/0 conn(0x7feb3006bc50 0x7feb300ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.378 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb38106640 1 --2- 172.21.15.122:0/494171801 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb3006de90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.378 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb38106640 1 -- 172.21.15.122:0/494171801 >> 172.21.15.122:0/494171801 conn(0x7feb300b1af0 msgr2=0x7feb30070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:19.378 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb38106640 1 -- 172.21.15.122:0/494171801 shutdown_connections 2023-12-16T21:57:19.378 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.373+0000 7feb38106640 1 -- 172.21.15.122:0/494171801 wait complete. 2023-12-16T21:57:19.379 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 Processor -- start 2023-12-16T21:57:19.379 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 -- start start 2023-12-16T21:57:19.380 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb3006bc50 0x7feb3014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300aeb80 0x7feb30152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb30153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3014c590 con 0x7feb300aeb80 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300b3e00 con 0x7feb3006bc50 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300b31f0 con 0x7feb30154580 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb35ea2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb3006bc50 0x7feb3014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb366a3640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb30153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb35ea2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb3006bc50 0x7feb3014b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:59974/0 (socket says 172.21.15.122:59974) 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb35ea2640 1 -- 172.21.15.122:0/1390471788 learned_addr learned my addr 172.21.15.122:0/1390471788 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb356a1640 1 --2- 172.21.15.122:0/1390471788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300aeb80 0x7feb30152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb366a3640 1 -- 172.21.15.122:0/1390471788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb3006bc50 msgr2=0x7feb3014b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb366a3640 1 --2- 172.21.15.122:0/1390471788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb3006bc50 0x7feb3014b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.381 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb366a3640 1 -- 172.21.15.122:0/1390471788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300aeb80 msgr2=0x7feb30152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb366a3640 1 --2- 172.21.15.122:0/1390471788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300aeb80 0x7feb30152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb366a3640 1 -- 172.21.15.122:0/1390471788 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c009420 con 0x7feb30154580 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb366a3640 1 --2- 172.21.15.122:0/1390471788 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb30153110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb28007d90 tx=0x7feb2800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb267fc640 1 -- 172.21.15.122:0/1390471788 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28018070 con 0x7feb30154580 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb267fc640 1 -- 172.21.15.122:0/1390471788 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2800edd0 con 0x7feb30154580 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb300b34d0 con 0x7feb30154580 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb267fc640 1 -- 172.21.15.122:0/1390471788 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28003bc0 con 0x7feb30154580 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb300b3a90 con 0x7feb30154580 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb267fc640 1 -- 172.21.15.122:0/1390471788 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feb28014720 con 0x7feb30154580 2023-12-16T21:57:19.382 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb267fc640 1 -- 172.21.15.122:0/1390471788 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7feb2801d8f0 con 0x7feb30154580 2023-12-16T21:57:19.383 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.377+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb30072060 con 0x7feb30154580 2023-12-16T21:57:19.394 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.393+0000 7feb267fc640 1 -- 172.21.15.122:0/1390471788 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7feb2801d410 con 0x7feb30154580 2023-12-16T21:57:19.462 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.457+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "osd new", "uuid": "814fdfdf-9ed9-440c-814d-7d73b1a21072", "id": 0} v 0) v1 -- 0x7feb300b3f70 con 0x7feb30154580 2023-12-16T21:57:19.470 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.465+0000 7feb267fc640 1 -- 172.21.15.122:0/1390471788 <== mon.2 v2:172.21.15.122:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "814fdfdf-9ed9-440c-814d-7d73b1a21072", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7feb28014bb0 con 0x7feb30154580 2023-12-16T21:57:19.470 INFO:teuthology.orchestra.run.smithi122.stdout:0 2023-12-16T21:57:19.471 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 msgr2=0x7feb30153110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.471 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 --2- 172.21.15.122:0/1390471788 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb30153110 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb28007d90 tx=0x7feb2800e4b0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.472 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 shutdown_connections 2023-12-16T21:57:19.472 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 --2- 172.21.15.122:0/1390471788 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feb30154580 0x7feb30153110 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.472 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 --2- 172.21.15.122:0/1390471788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300aeb80 0x7feb30152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.472 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 --2- 172.21.15.122:0/1390471788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feb3006bc50 0x7feb3014b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.472 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 >> 172.21.15.122:0/1390471788 conn(0x7feb300b1af0 msgr2=0x7feb30158f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:19.472 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 shutdown_connections 2023-12-16T21:57:19.472 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.469+0000 7feb38106640 1 -- 172.21.15.122:0/1390471788 wait complete. 2023-12-16T21:57:19.485 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:19.481+0000 7fca5fa10180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T21:57:19.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 8f53d5ae-12c2-4944-9728-b31af8119226 1 2023-12-16T21:57:19.497 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:19.498+0000 7ff19134b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T21:57:19.597 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.593+0000 7f5fc94e6640 1 Processor -- start 2023-12-16T21:57:19.597 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.593+0000 7f5fc94e6640 1 -- start start 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.593+0000 7f5fc94e6640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5fc4143b30 0x7f5fc4072360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.593+0000 7f5fc94e6640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc40728a0 0x7f5fc4071590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.593+0000 7f5fc94e6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5fc4071ad0 0x7f5fc415db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.593+0000 7f5fc94e6640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc4071f50 con 0x7f5fc4143b30 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.593+0000 7f5fc94e6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc406ff20 con 0x7f5fc4071ad0 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.593+0000 7f5fc94e6640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40cec90 con 0x7f5fc40728a0 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5fc4071ad0 0x7f5fc415db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fbbfff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc40728a0 0x7f5fc4071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5fc4071ad0 0x7f5fc415db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40364/0 (socket says 172.21.15.122:40364) 2023-12-16T21:57:19.599 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc37fe640 1 -- 172.21.15.122:0/3135257923 learned_addr learned my addr 172.21.15.122:0/3135257923 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc2ffd640 1 --2- 172.21.15.122:0/3135257923 >> v2:172.21.15.28:3300/0 conn(0x7f5fc4143b30 0x7f5fc4072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fbbfff640 1 -- 172.21.15.122:0/3135257923 >> v2:172.21.15.122:3300/0 conn(0x7f5fc4071ad0 msgr2=0x7f5fc415db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fbbfff640 1 --2- 172.21.15.122:0/3135257923 >> v2:172.21.15.122:3300/0 conn(0x7f5fc4071ad0 0x7f5fc415db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fbbfff640 1 -- 172.21.15.122:0/3135257923 >> v2:172.21.15.28:3300/0 conn(0x7f5fc4143b30 msgr2=0x7f5fc4072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fbbfff640 1 --2- 172.21.15.122:0/3135257923 >> v2:172.21.15.28:3300/0 conn(0x7f5fc4143b30 0x7f5fc4072360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fbbfff640 1 -- 172.21.15.122:0/3135257923 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc40cef70 con 0x7f5fc40728a0 2023-12-16T21:57:19.600 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc37fe640 1 --2- 172.21.15.122:0/3135257923 >> v2:172.21.15.122:3300/0 conn(0x7f5fc4071ad0 0x7f5fc415db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:19.601 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fbbfff640 1 --2- 172.21.15.122:0/3135257923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc40728a0 0x7f5fc4071590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4071280 tx=0x7f5fac01e840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=921e867b5ba673a0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:19.602 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc27fc640 1 -- 172.21.15.122:0/3135257923 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac0a78d0 con 0x7f5fc40728a0 2023-12-16T21:57:19.602 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc27fc640 1 -- 172.21.15.122:0/3135257923 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fac0a7a30 con 0x7f5fc40728a0 2023-12-16T21:57:19.602 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc27fc640 1 -- 172.21.15.122:0/3135257923 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac0a7dc0 con 0x7f5fc40728a0 2023-12-16T21:57:19.602 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3135257923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc40728a0 msgr2=0x7f5fc4071590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.603 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc94e6640 1 --2- 172.21.15.122:0/3135257923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc40728a0 0x7f5fc4071590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4071280 tx=0x7f5fac01e840 comp rx=0 tx=0).stop 2023-12-16T21:57:19.603 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3135257923 shutdown_connections 2023-12-16T21:57:19.603 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.597+0000 7f5fc94e6640 1 --2- 172.21.15.122:0/3135257923 >> v2:172.21.15.122:3300/0 conn(0x7f5fc4071ad0 0x7f5fc415db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.603 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 --2- 172.21.15.122:0/3135257923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc40728a0 0x7f5fc4071590 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.603 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 --2- 172.21.15.122:0/3135257923 >> v2:172.21.15.28:3300/0 conn(0x7f5fc4143b30 0x7f5fc4072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.603 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3135257923 >> 172.21.15.122:0/3135257923 conn(0x7f5fc406d510 msgr2=0x7f5fc406d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:19.603 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3135257923 shutdown_connections 2023-12-16T21:57:19.603 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3135257923 wait complete. 2023-12-16T21:57:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 Processor -- start 2023-12-16T21:57:19.604 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- start start 2023-12-16T21:57:19.606 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fc4071ad0 0x7f5fc41634c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.606 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fc4143b30 0x7f5fc415ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc413b360 0x7f5fc4142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc415f180 con 0x7f5fc4071ad0 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc4164c40 con 0x7f5fc4143b30 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc41647b0 con 0x7f5fc413b360 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fc4143b30 0x7f5fc415ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc37fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc413b360 0x7f5fc4142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbbfff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fc4143b30 0x7f5fc415ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40374/0 (socket says 172.21.15.122:40374) 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc37fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc413b360 0x7f5fc4142750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.122:50042/0 (socket says 172.21.15.122:50042) 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbbfff640 1 -- 172.21.15.122:0/3734185968 learned_addr learned my addr 172.21.15.122:0/3734185968 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc2ffd640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fc4071ad0 0x7f5fc41634c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc37fe640 1 -- 172.21.15.122:0/3734185968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fc4143b30 msgr2=0x7f5fc415ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc37fe640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fc4143b30 0x7f5fc415ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc37fe640 1 -- 172.21.15.122:0/3734185968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fc4071ad0 msgr2=0x7f5fc41634c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc37fe640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fc4071ad0 0x7f5fc41634c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.607 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc37fe640 1 -- 172.21.15.122:0/3734185968 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc009b30 con 0x7f5fc413b360 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbbfff640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fc4143b30 0x7f5fc415ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc37fe640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc413b360 0x7f5fc4142750 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00eed0 tx=0x7f5fbc00b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbb7fe640 1 -- 172.21.15.122:0/3734185968 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc018070 con 0x7f5fc413b360 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbb7fe640 1 -- 172.21.15.122:0/3734185968 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc00bb10 con 0x7f5fc413b360 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac00b6c0 con 0x7f5fc413b360 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbb7fe640 1 -- 172.21.15.122:0/3734185968 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc010040 con 0x7f5fc413b360 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc4163e80 con 0x7f5fc413b360 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbb7fe640 1 -- 172.21.15.122:0/3734185968 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5fbc003ca0 con 0x7f5fc413b360 2023-12-16T21:57:19.608 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.601+0000 7f5fbb7fe640 1 -- 172.21.15.122:0/3734185968 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f5fbc01d7c0 con 0x7f5fc413b360 2023-12-16T21:57:19.609 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.605+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc40b5340 con 0x7f5fc413b360 2023-12-16T21:57:19.609 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.605+0000 7f5fbb7fe640 1 -- 172.21.15.122:0/3734185968 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f5fbc004700 con 0x7f5fc413b360 2023-12-16T21:57:19.665 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:19.661+0000 7fca5fa10180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T21:57:19.672 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.669+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8f53d5ae-12c2-4944-9728-b31af8119226", "id": 1} v 0) v1 -- 0x7f5fc40674d0 con 0x7f5fc413b360 2023-12-16T21:57:19.678 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:19.678+0000 7ff19134b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T21:57:19.681 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.677+0000 7f5fbb7fe640 1 -- 172.21.15.122:0/3734185968 <== mon.2 v2:172.21.15.122:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f53d5ae-12c2-4944-9728-b31af8119226", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5fbc02b0b0 con 0x7f5fc413b360 2023-12-16T21:57:19.681 INFO:teuthology.orchestra.run.smithi122.stdout:1 2023-12-16T21:57:19.682 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.677+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc413b360 msgr2=0x7f5fc4142750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.682 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.677+0000 7f5fc94e6640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc413b360 0x7f5fc4142750 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00eed0 tx=0x7f5fbc00b480 comp rx=0 tx=0).stop 2023-12-16T21:57:19.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.677+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 shutdown_connections 2023-12-16T21:57:19.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.677+0000 7f5fc94e6640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fc413b360 0x7f5fc4142750 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.677+0000 7f5fc94e6640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fc4143b30 0x7f5fc415ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.681+0000 7f5fc94e6640 1 --2- 172.21.15.122:0/3734185968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fc4071ad0 0x7f5fc41634c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.681+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 >> 172.21.15.122:0/3734185968 conn(0x7f5fc406d510 msgr2=0x7f5fc415c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:19.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.681+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 shutdown_connections 2023-12-16T21:57:19.683 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.681+0000 7f5fc94e6640 1 -- 172.21.15.122:0/3734185968 wait complete. 2023-12-16T21:57:19.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 1804f849-d8f5-4fbb-a6e1-6b7572c80d1a 2 2023-12-16T21:57:19.813 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.809+0000 7f432523b640 1 Processor -- start 2023-12-16T21:57:19.813 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.809+0000 7f432523b640 1 -- start start 2023-12-16T21:57:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f432013f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432006ea50 con 0x7f432006f7f0 2023-12-16T21:57:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4320140380 con 0x7f432013cd60 2023-12-16T21:57:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43200cf8f0 con 0x7f4320071c30 2023-12-16T21:57:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431ed9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f432013cd60 0x7f4320071520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58556/0 (socket says 172.21.15.122:58556) 2023-12-16T21:57:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431ed9d640 1 -- 172.21.15.122:0/1839695852 learned_addr learned my addr 172.21.15.122:0/1839695852 (peer_addr_for_me v1:172.21.15.122:0/0) 2023-12-16T21:57:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431e59c640 1 --2- 172.21.15.122:0/1839695852 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f432013f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.815 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431dd9b640 1 -- 172.21.15.122:0/1839695852 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361197938 0 0) 0x7f4320140380 con 0x7f432013cd60 2023-12-16T21:57:19.816 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431dd9b640 1 -- 172.21.15.122:0/1839695852 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f430c003530 con 0x7f432013cd60 2023-12-16T21:57:19.816 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431e59c640 1 -- 172.21.15.122:0/1839695852 >> v1:172.21.15.122:6789/0 conn(0x7f432013cd60 legacy=0x7f4320071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.816 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431e59c640 1 -- 172.21.15.122:0/1839695852 >> v1:172.21.15.28:6789/0 conn(0x7f432006f7f0 legacy=0x7f432013fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.816 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431e59c640 1 -- 172.21.15.122:0/1839695852 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432006ed10 con 0x7f4320071c30 2023-12-16T21:57:19.817 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431e59c640 1 --2- 172.21.15.122:0/1839695852 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f432013f630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43080097f0 tx=0x7f430801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe946d858af71202 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:19.817 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431dd9b640 1 -- 172.21.15.122:0/1839695852 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43080aa5f0 con 0x7f4320071c30 2023-12-16T21:57:19.817 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431dd9b640 1 -- 172.21.15.122:0/1839695852 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43080aa750 con 0x7f4320071c30 2023-12-16T21:57:19.817 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f431dd9b640 1 -- 172.21.15.122:0/1839695852 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43080aaae0 con 0x7f4320071c30 2023-12-16T21:57:19.818 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 -- 172.21.15.122:0/1839695852 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 msgr2=0x7f432013f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.818 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 --2- 172.21.15.122:0/1839695852 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f432013f630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43080097f0 tx=0x7f430801c490 comp rx=0 tx=0).stop 2023-12-16T21:57:19.819 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 -- 172.21.15.122:0/1839695852 shutdown_connections 2023-12-16T21:57:19.819 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 --2- 172.21.15.122:0/1839695852 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f432013f630 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.819 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 -- 172.21.15.122:0/1839695852 >> 172.21.15.122:0/1839695852 conn(0x7f43200b1db0 msgr2=0x7f43200b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:19.819 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 -- 172.21.15.122:0/1839695852 shutdown_connections 2023-12-16T21:57:19.819 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.813+0000 7f432523b640 1 -- 172.21.15.122:0/1839695852 wait complete. 2023-12-16T21:57:19.821 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 Processor -- start 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 -- start start 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432006f7f0 0x7f43201384b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f4320137da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f432013cd60 0x7f4320159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43201382e0 con 0x7f432006f7f0 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4320158a30 con 0x7f432013cd60 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4320158d10 con 0x7f4320071c30 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431f59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f432013cd60 0x7f4320159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431e59c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f4320137da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431e59c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f4320137da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.122:50066/0 (socket says 172.21.15.122:50066) 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431f59e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f432013cd60 0x7f4320159ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40386/0 (socket says 172.21.15.122:40386) 2023-12-16T21:57:19.822 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431e59c640 1 -- 172.21.15.122:0/3655208098 learned_addr learned my addr 172.21.15.122:0/3655208098 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431ed9d640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432006f7f0 0x7f43201384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431f59e640 1 -- 172.21.15.122:0/3655208098 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 msgr2=0x7f4320137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431f59e640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f4320137da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431f59e640 1 -- 172.21.15.122:0/3655208098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432006f7f0 msgr2=0x7f43201384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431f59e640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432006f7f0 0x7f43201384b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431f59e640 1 -- 172.21.15.122:0/3655208098 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308009420 con 0x7f432013cd60 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431ed9d640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432006f7f0 0x7f43201384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431e59c640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f4320137da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f431f59e640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f432013cd60 0x7f4320159ad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4320072940 tx=0x7f431000cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f43177fe640 1 -- 172.21.15.122:0/3655208098 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310003b80 con 0x7f432013cd60 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f43177fe640 1 -- 172.21.15.122:0/3655208098 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4310003ce0 con 0x7f432013cd60 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4320158ff0 con 0x7f432013cd60 2023-12-16T21:57:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f43177fe640 1 -- 172.21.15.122:0/3655208098 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310005310 con 0x7f432013cd60 2023-12-16T21:57:19.824 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43201595b0 con 0x7f432013cd60 2023-12-16T21:57:19.824 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f43177fe640 1 -- 172.21.15.122:0/3655208098 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f431000a5e0 con 0x7f432013cd60 2023-12-16T21:57:19.824 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f43177fe640 1 -- 172.21.15.122:0/3655208098 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f43100129d0 con 0x7f432013cd60 2023-12-16T21:57:19.824 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.817+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43200674d0 con 0x7f432013cd60 2023-12-16T21:57:19.824 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.821+0000 7f43177fe640 1 -- 172.21.15.122:0/3655208098 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f4310012c30 con 0x7f432013cd60 2023-12-16T21:57:19.889 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.885+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1804f849-d8f5-4fbb-a6e1-6b7572c80d1a", "id": 2} v 0) v1 -- 0x7f4320072550 con 0x7f432013cd60 2023-12-16T21:57:19.900 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f43177fe640 1 -- 172.21.15.122:0/3655208098 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1804f849-d8f5-4fbb-a6e1-6b7572c80d1a", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f43100125b0 con 0x7f432013cd60 2023-12-16T21:57:19.901 INFO:teuthology.orchestra.run.smithi122.stdout:2 2023-12-16T21:57:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f432013cd60 msgr2=0x7f4320159ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:19.901 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f432013cd60 0x7f4320159ad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4320072940 tx=0x7f431000cb90 comp rx=0 tx=0).stop 2023-12-16T21:57:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 shutdown_connections 2023-12-16T21:57:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f432013cd60 0x7f4320159ad0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4320071c30 0x7f4320137da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 --2- 172.21.15.122:0/3655208098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432006f7f0 0x7f43201384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 >> 172.21.15.122:0/3655208098 conn(0x7f43200b1db0 msgr2=0x7f4320139b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 shutdown_connections 2023-12-16T21:57:19.902 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:19.897+0000 7f432523b640 1 -- 172.21.15.122:0/3655208098 wait complete. 2023-12-16T21:57:19.917 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 98e0c424-17ea-4566-a959-48efa476f1a1 3 2023-12-16T21:57:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a414c640 1 Processor -- start 2023-12-16T21:57:20.033 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a414c640 1 -- start start 2023-12-16T21:57:20.036 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a414c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc29c13cff0 0x7fc29c0cc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.036 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a414c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0cba80 con 0x7fc29c15e170 2023-12-16T21:57:20.036 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a414c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c15e750 con 0x7fc29c06b540 2023-12-16T21:57:20.036 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a414c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c15ea30 con 0x7fc29c13cff0 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a16e7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc29c06b540 0x7fc29c15da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58568/0 (socket says 172.21.15.122:58568) 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a16e7640 1 -- 172.21.15.122:0/3259916811 learned_addr learned my addr 172.21.15.122:0/3259916811 (peer_addr_for_me v1:172.21.15.122:0/0) 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a1ee8640 1 --2- 172.21.15.122:0/3259916811 >> v2:172.21.15.28:3300/0 conn(0x7fc29c13cff0 0x7fc29c0cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359360570 0 0) 0x7fc29c15e750 con 0x7fc29c06b540 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc288003530 con 0x7fc29c06b540 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1522154654 0 0) 0x7fc288003530 con 0x7fc29c06b540 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 >> v2:172.21.15.28:3300/0 conn(0x7fc29c13cff0 msgr2=0x7fc29c0cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 --2- 172.21.15.122:0/3259916811 >> v2:172.21.15.28:3300/0 conn(0x7fc29c13cff0 0x7fc29c0cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 >> v1:172.21.15.28:6789/0 conn(0x7fc29c15e170 legacy=0x7fc29c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c0cbd20 con 0x7fc29c06b540 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fc28c001680 con 0x7fc29c06b540 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 >> v1:172.21.15.122:6789/0 conn(0x7fc29c06b540 legacy=0x7fc29c15da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.037 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc288003980 0x7fc288003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc288004380 0x7fc288004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2880050f0 0x7fc28800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c15ea30 con 0x7fc2880050f0 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28800c030 con 0x7fc288003980 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.029+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28800c1f0 con 0x7fc288004380 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a1ee8640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc288003980 0x7fc288003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a16e7640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc288004380 0x7fc288004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a26e9640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2880050f0 0x7fc28800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a1ee8640 1 -- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc288004380 msgr2=0x7fc288004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a1ee8640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc288004380 0x7fc288004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a1ee8640 1 -- 172.21.15.122:0/3259916811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2880050f0 msgr2=0x7fc28800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a1ee8640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2880050f0 0x7fc28800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.038 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a1ee8640 1 -- 172.21.15.122:0/3259916811 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc288003530 con 0x7fc288003980 2023-12-16T21:57:20.039 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a26e9640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2880050f0 0x7fc28800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:20.039 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a26e9640 1 -- 172.21.15.122:0/3259916811 reap_dead start 2023-12-16T21:57:20.044 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a1ee8640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc288003980 0x7fc288003d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc294009fc0 tx=0x7fc29401cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ea6c0af50f321d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:20.044 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2940a5a10 con 0x7fc288003980 2023-12-16T21:57:20.044 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29400f040 con 0x7fc288003980 2023-12-16T21:57:20.044 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a0ee6640 1 -- 172.21.15.122:0/3259916811 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2940a5db0 con 0x7fc288003980 2023-12-16T21:57:20.044 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a414c640 1 -- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc288003980 msgr2=0x7fc288003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.044 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a414c640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc288003980 0x7fc288003d60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc294009fc0 tx=0x7fc29401cc60 comp rx=0 tx=0).stop 2023-12-16T21:57:20.044 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a414c640 1 -- 172.21.15.122:0/3259916811 shutdown_connections 2023-12-16T21:57:20.045 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a414c640 1 --2- 172.21.15.122:0/3259916811 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc288003980 0x7fc288003d60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.045 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a414c640 1 -- 172.21.15.122:0/3259916811 >> 172.21.15.122:0/3259916811 conn(0x7fc29c0710b0 msgr2=0x7fc29c0714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:20.045 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a414c640 1 -- 172.21.15.122:0/3259916811 shutdown_connections 2023-12-16T21:57:20.045 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.033+0000 7fc2a414c640 1 -- 172.21.15.122:0/3259916811 wait complete. 2023-12-16T21:57:20.045 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 Processor -- start 2023-12-16T21:57:20.045 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 -- start start 2023-12-16T21:57:20.045 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc29c06e570 0x7fc29c141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29c06e950 0x7fc29c1f9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29c1f9550 0x7fc29c156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c133880 con 0x7fc29c06e950 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1332e0 con 0x7fc29c1f9550 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c132760 con 0x7fc29c06e570 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc29c06e570 0x7fc29c141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a26e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29c1f9550 0x7fc29c156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc29c06e570 0x7fc29c141f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.122:50090/0 (socket says 172.21.15.122:50090) 2023-12-16T21:57:20.046 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 -- 172.21.15.122:0/1559471980 learned_addr learned my addr 172.21.15.122:0/1559471980 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a16e7640 1 --2- 172.21.15.122:0/1559471980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29c06e950 0x7fc29c1f9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 -- 172.21.15.122:0/1559471980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29c1f9550 msgr2=0x7fc29c156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 --2- 172.21.15.122:0/1559471980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29c1f9550 0x7fc29c156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 -- 172.21.15.122:0/1559471980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29c06e950 msgr2=0x7fc29c1f9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 --2- 172.21.15.122:0/1559471980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29c06e950 0x7fc29c1f9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 -- 172.21.15.122:0/1559471980 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c132a40 con 0x7fc29c06e570 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a1ee8640 1 --2- 172.21.15.122:0/1559471980 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc29c06e570 0x7fc29c141f10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc29401ca90 tx=0x7fc294009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2927fc640 1 -- 172.21.15.122:0/1559471980 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2940039a0 con 0x7fc29c06e570 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2927fc640 1 -- 172.21.15.122:0/1559471980 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc294003b00 con 0x7fc29c06e570 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c132d20 con 0x7fc29c06e570 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2927fc640 1 -- 172.21.15.122:0/1559471980 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294003e70 con 0x7fc29c06e570 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c1f9a00 con 0x7fc29c06e570 2023-12-16T21:57:20.047 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2927fc640 1 -- 172.21.15.122:0/1559471980 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc2940b4040 con 0x7fc29c06e570 2023-12-16T21:57:20.048 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2927fc640 1 -- 172.21.15.122:0/1559471980 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fc2940adba0 con 0x7fc29c06e570 2023-12-16T21:57:20.048 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.037+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c132760 con 0x7fc29c06e570 2023-12-16T21:57:20.048 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.041+0000 7fc2927fc640 1 -- 172.21.15.122:0/1559471980 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7fc2940c3060 con 0x7fc29c06e570 2023-12-16T21:57:20.110 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.105+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "osd new", "uuid": "98e0c424-17ea-4566-a959-48efa476f1a1", "id": 3} v 0) v1 -- 0x7fc29c1f67b0 con 0x7fc29c06e570 2023-12-16T21:57:20.115 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2927fc640 1 -- 172.21.15.122:0/1559471980 <== mon.2 v2:172.21.15.122:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98e0c424-17ea-4566-a959-48efa476f1a1", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc2940b2070 con 0x7fc29c06e570 2023-12-16T21:57:20.115 INFO:teuthology.orchestra.run.smithi122.stdout:3 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc29c06e570 msgr2=0x7fc29c141f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 --2- 172.21.15.122:0/1559471980 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc29c06e570 0x7fc29c141f10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc29401ca90 tx=0x7fc294009750 comp rx=0 tx=0).stop 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 shutdown_connections 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 --2- 172.21.15.122:0/1559471980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc29c1f9550 0x7fc29c156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 --2- 172.21.15.122:0/1559471980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29c06e950 0x7fc29c1f9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 --2- 172.21.15.122:0/1559471980 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc29c06e570 0x7fc29c141f10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 >> 172.21.15.122:0/1559471980 conn(0x7fc29c0710b0 msgr2=0x7fc29c158fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 shutdown_connections 2023-12-16T21:57:20.117 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.113+0000 7fc2a414c640 1 -- 172.21.15.122:0/1559471980 wait complete. 2023-12-16T21:57:20.131 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 2585d9a5-cc84-4342-9684-4eda8e1d7513 4 2023-12-16T21:57:20.245 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd97d9640 1 Processor -- start 2023-12-16T21:57:20.245 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd97d9640 1 -- start start 2023-12-16T21:57:20.246 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd97d9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd40cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd97d9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcdd40ceae0 0x7fcdd406e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd97d9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcdd406ed70 0x7fcdd406fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd97d9640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd406f200 con 0x7fcdd40ceae0 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd97d9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd40cfbd0 con 0x7fcdd406ed70 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd97d9640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd40cefc0 con 0x7fcdd4144830 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd40cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd37fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcdd406ed70 0x7fcdd406fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd40cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.122:50094/0 (socket says 172.21.15.122:50094) 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd2ffd640 1 -- 172.21.15.122:0/3127830604 learned_addr learned my addr 172.21.15.122:0/3127830604 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdcbfff640 1 --2- 172.21.15.122:0/3127830604 >> v2:172.21.15.28:3300/0 conn(0x7fcdd40ceae0 0x7fcdd406e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.247 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd37fe640 1 -- 172.21.15.122:0/3127830604 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 msgr2=0x7fcdd40cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:20.248 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd37fe640 1 --2- 172.21.15.122:0/3127830604 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd40cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.248 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd37fe640 1 -- 172.21.15.122:0/3127830604 >> v2:172.21.15.28:3300/0 conn(0x7fcdd40ceae0 msgr2=0x7fcdd406e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:20.248 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd37fe640 1 --2- 172.21.15.122:0/3127830604 >> v2:172.21.15.28:3300/0 conn(0x7fcdd40ceae0 0x7fcdd406e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.248 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd37fe640 1 -- 172.21.15.122:0/3127830604 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd40cf2a0 con 0x7fcdd406ed70 2023-12-16T21:57:20.248 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.241+0000 7fcdd2ffd640 1 --2- 172.21.15.122:0/3127830604 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd40cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:20.249 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd37fe640 1 --2- 172.21.15.122:0/3127830604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd406fd70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40cce40 tx=0x7fcdcc01f0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6dad1b01492a3b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:20.249 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd27fc640 1 -- 172.21.15.122:0/3127830604 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0b0440 con 0x7fcdd406ed70 2023-12-16T21:57:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd27fc640 1 -- 172.21.15.122:0/3127830604 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc0b05a0 con 0x7fcdd406ed70 2023-12-16T21:57:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd27fc640 1 -- 172.21.15.122:0/3127830604 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0b0930 con 0x7fcdd406ed70 2023-12-16T21:57:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 -- 172.21.15.122:0/3127830604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 msgr2=0x7fcdd406fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.250 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 --2- 172.21.15.122:0/3127830604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd406fd70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40cce40 tx=0x7fcdcc01f0f0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 -- 172.21.15.122:0/3127830604 shutdown_connections 2023-12-16T21:57:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 --2- 172.21.15.122:0/3127830604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd406fd70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 --2- 172.21.15.122:0/3127830604 >> v2:172.21.15.28:3300/0 conn(0x7fcdd40ceae0 0x7fcdd406e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 --2- 172.21.15.122:0/3127830604 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd40cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 -- 172.21.15.122:0/3127830604 >> 172.21.15.122:0/3127830604 conn(0x7fcdd40d0f10 msgr2=0x7fcdd40d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 -- 172.21.15.122:0/3127830604 shutdown_connections 2023-12-16T21:57:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.245+0000 7fcdd97d9640 1 -- 172.21.15.122:0/3127830604 wait complete. 2023-12-16T21:57:20.251 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 Processor -- start 2023-12-16T21:57:20.252 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 -- start start 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd413ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd40ceae0 0x7fcdd415d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd415d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd413c1a0 con 0x7fcdd40ceae0 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd415dad0 con 0x7fcdd406ed70 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd413d200 con 0x7fcdd4144830 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd413ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd413ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40418/0 (socket says 172.21.15.122:40418) 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd37fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd415d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 -- 172.21.15.122:0/37942033 learned_addr learned my addr 172.21.15.122:0/37942033 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:20.254 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdcbfff640 1 --2- 172.21.15.122:0/37942033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd40ceae0 0x7fcdd415d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 -- 172.21.15.122:0/37942033 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 msgr2=0x7fcdd415d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 --2- 172.21.15.122:0/37942033 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd415d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 -- 172.21.15.122:0/37942033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd40ceae0 msgr2=0x7fcdd415d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 --2- 172.21.15.122:0/37942033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd40ceae0 0x7fcdd415d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 -- 172.21.15.122:0/37942033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc40094d0 con 0x7fcdd406ed70 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd2ffd640 1 --2- 172.21.15.122:0/37942033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd413ebf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e930 tx=0x7fcdc400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdcb7fe640 1 -- 172.21.15.122:0/37942033 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400ca00 con 0x7fcdd406ed70 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdcb7fe640 1 -- 172.21.15.122:0/37942033 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc400cb60 con 0x7fcdd406ed70 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdcc009a70 con 0x7fcdd406ed70 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdcb7fe640 1 -- 172.21.15.122:0/37942033 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4013470 con 0x7fcdd406ed70 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd413c720 con 0x7fcdd406ed70 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdcb7fe640 1 -- 172.21.15.122:0/37942033 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcdc4005ca0 con 0x7fcdd406ed70 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdcb7fe640 1 -- 172.21.15.122:0/37942033 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fcdc40067f0 con 0x7fcdd406ed70 2023-12-16T21:57:20.255 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.249+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd40674d0 con 0x7fcdd406ed70 2023-12-16T21:57:20.256 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.253+0000 7fcdcb7fe640 1 -- 172.21.15.122:0/37942033 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7fcdc4021060 con 0x7fcdd406ed70 2023-12-16T21:57:20.270 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.270+0000 7ff19134b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T21:57:20.292 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.289+0000 7fca5fa10180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T21:57:20.323 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.321+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2585d9a5-cc84-4342-9684-4eda8e1d7513", "id": 4} v 0) v1 -- 0x7fcdd40cb2d0 con 0x7fcdd406ed70 2023-12-16T21:57:20.325 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.326+0000 7ff19134b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T21:57:20.329 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdcb7fe640 1 -- 172.21.15.122:0/37942033 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2585d9a5-cc84-4342-9684-4eda8e1d7513", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcdc4029020 con 0x7fcdd406ed70 2023-12-16T21:57:20.329 INFO:teuthology.orchestra.run.smithi122.stdout:4 2023-12-16T21:57:20.330 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 msgr2=0x7fcdd413ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.330 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 --2- 172.21.15.122:0/37942033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd413ebf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400e930 tx=0x7fcdc400edf0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.330 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 shutdown_connections 2023-12-16T21:57:20.330 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 --2- 172.21.15.122:0/37942033 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcdd4144830 0x7fcdd415d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.330 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 --2- 172.21.15.122:0/37942033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdd40ceae0 0x7fcdd415d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.331 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 --2- 172.21.15.122:0/37942033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdd406ed70 0x7fcdd413ebf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.331 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 >> 172.21.15.122:0/37942033 conn(0x7fcdd40d0f10 msgr2=0x7fcdd40d12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:20.331 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 shutdown_connections 2023-12-16T21:57:20.331 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.325+0000 7fcdd97d9640 1 -- 172.21.15.122:0/37942033 wait complete. 2023-12-16T21:57:20.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new a2afbca9-85c5-4583-9cbf-33cbf0a54677 5 2023-12-16T21:57:20.350 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.345+0000 7fca5fa10180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T21:57:20.381 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.378+0000 7ff19134b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T21:57:20.404 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.401+0000 7fca5fa10180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T21:57:20.449 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.445+0000 7f38f4279640 1 Processor -- start 2023-12-16T21:57:20.449 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.445+0000 7f38f4279640 1 -- start start 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.445+0000 7f38f4279640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec0cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.445+0000 7f38f4279640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f38ec06ef40 0x7f38ec0b4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.445+0000 7f38f4279640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ec0cfbd0 con 0x7f38ec0ceae0 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.445+0000 7f38f4279640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ec0cefc0 con 0x7f38ec06ef40 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.445+0000 7f38f4279640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0cf2a0 con 0x7f38ec144830 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2816640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f38ec06ef40 0x7f38ec0b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2015640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec0cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2816640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f38ec06ef40 0x7f38ec0b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40426/0 (socket says 172.21.15.122:40426) 2023-12-16T21:57:20.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2816640 1 -- 172.21.15.122:0/1073862525 learned_addr learned my addr 172.21.15.122:0/1073862525 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:20.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2816640 1 -- 172.21.15.122:0/1073862525 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 msgr2=0x7f38ec0cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:20.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2816640 1 --2- 172.21.15.122:0/1073862525 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec0cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2816640 1 -- 172.21.15.122:0/1073862525 >> v1:172.21.15.122:6789/0 conn(0x7f38ec0ceae0 legacy=0x7f38ec06e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2816640 1 -- 172.21.15.122:0/1073862525 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec0cf580 con 0x7f38ec06ef40 2023-12-16T21:57:20.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2015640 1 --2- 172.21.15.122:0/1073862525 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:20.453 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f2816640 1 --2- 172.21.15.122:0/1073862525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0b4050 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009e80 tx=0x7f38e401f1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28144bc8f8a66f83 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:20.453 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f1013640 1 -- 172.21.15.122:0/1073862525 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e40af400 con 0x7f38ec06ef40 2023-12-16T21:57:20.453 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f1013640 1 -- 172.21.15.122:0/1073862525 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e40ab050 con 0x7f38ec06ef40 2023-12-16T21:57:20.453 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f1013640 1 -- 172.21.15.122:0/1073862525 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e40af780 con 0x7f38ec06ef40 2023-12-16T21:57:20.454 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f4279640 1 -- 172.21.15.122:0/1073862525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 msgr2=0x7f38ec0b4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.454 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f4279640 1 --2- 172.21.15.122:0/1073862525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0b4050 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009e80 tx=0x7f38e401f1e0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.454 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f4279640 1 -- 172.21.15.122:0/1073862525 shutdown_connections 2023-12-16T21:57:20.454 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f4279640 1 --2- 172.21.15.122:0/1073862525 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0b4050 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.454 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f4279640 1 --2- 172.21.15.122:0/1073862525 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec0cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.454 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f4279640 1 -- 172.21.15.122:0/1073862525 >> 172.21.15.122:0/1073862525 conn(0x7f38ec0d0f10 msgr2=0x7f38ec0d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:20.454 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f4279640 1 -- 172.21.15.122:0/1073862525 shutdown_connections 2023-12-16T21:57:20.454 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.449+0000 7f38f4279640 1 -- 172.21.15.122:0/1073862525 wait complete. 2023-12-16T21:57:20.455 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 Processor -- start 2023-12-16T21:57:20.455 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 -- start start 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ec0ceae0 0x7f38ec15d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec15dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0c8a30 con 0x7f38ec0ceae0 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0a8540 con 0x7f38ec06ef40 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0c8e50 con 0x7f38ec144830 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40434/0 (socket says 172.21.15.122:40434) 2023-12-16T21:57:20.457 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2816640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec15dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 -- 172.21.15.122:0/2996489213 learned_addr learned my addr 172.21.15.122:0/2996489213 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 -- 172.21.15.122:0/2996489213 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 msgr2=0x7f38ec15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 --2- 172.21.15.122:0/2996489213 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec15dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 -- 172.21.15.122:0/2996489213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ec0ceae0 msgr2=0x7f38ec15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 --2- 172.21.15.122:0/2996489213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ec0ceae0 0x7f38ec15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 -- 172.21.15.122:0/2996489213 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e8009420 con 0x7f38ec06ef40 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f2015640 1 --2- 172.21.15.122:0/2996489213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0c9e30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f38e800e9e0 tx=0x7f38e800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38e2ffd640 1 -- 172.21.15.122:0/2996489213 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e800ca30 con 0x7f38ec06ef40 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38e2ffd640 1 -- 172.21.15.122:0/2996489213 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e800cb90 con 0x7f38ec06ef40 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e4009b30 con 0x7f38ec06ef40 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38e2ffd640 1 -- 172.21.15.122:0/2996489213 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8013470 con 0x7f38ec06ef40 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ec0cd800 con 0x7f38ec06ef40 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38e2ffd640 1 -- 172.21.15.122:0/2996489213 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f38e8005ca0 con 0x7f38ec06ef40 2023-12-16T21:57:20.458 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38e2ffd640 1 -- 172.21.15.122:0/2996489213 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f38e8013b50 con 0x7f38ec06ef40 2023-12-16T21:57:20.459 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.453+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ec0674d0 con 0x7f38ec06ef40 2023-12-16T21:57:20.459 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.457+0000 7f38e2ffd640 1 -- 172.21.15.122:0/2996489213 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f38e8021060 con 0x7f38ec06ef40 2023-12-16T21:57:20.461 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.458+0000 7ff19134b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T21:57:20.485 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.481+0000 7fca5fa10180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T21:57:20.515 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.514+0000 7ff19134b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T21:57:20.523 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.521+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a2afbca9-85c5-4583-9cbf-33cbf0a54677", "id": 5} v 0) v1 -- 0x7f38ec0cd100 con 0x7f38ec06ef40 2023-12-16T21:57:20.528 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38e2ffd640 1 -- 172.21.15.122:0/2996489213 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a2afbca9-85c5-4583-9cbf-33cbf0a54677", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f38e8017070 con 0x7f38ec06ef40 2023-12-16T21:57:20.528 INFO:teuthology.orchestra.run.smithi122.stdout:5 2023-12-16T21:57:20.529 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 msgr2=0x7f38ec0c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.529 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 --2- 172.21.15.122:0/2996489213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0c9e30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f38e800e9e0 tx=0x7f38e800eea0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.529 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 shutdown_connections 2023-12-16T21:57:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 --2- 172.21.15.122:0/2996489213 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ec144830 0x7f38ec15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 --2- 172.21.15.122:0/2996489213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ec0ceae0 0x7f38ec15d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 --2- 172.21.15.122:0/2996489213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ec06ef40 0x7f38ec0c9e30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 >> 172.21.15.122:0/2996489213 conn(0x7f38ec0d0f10 msgr2=0x7f38ec0b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 shutdown_connections 2023-12-16T21:57:20.530 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.525+0000 7f38f4279640 1 -- 172.21.15.122:0/2996489213 wait complete. 2023-12-16T21:57:20.539 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.537+0000 7fca5fa10180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T21:57:20.544 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new b7d9aa20-7c15-425a-922b-659558ce1ef0 6 2023-12-16T21:57:20.595 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.594+0000 7ff19134b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T21:57:20.618 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.613+0000 7fca5fa10180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T21:57:20.650 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.650+0000 7ff19134b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T21:57:20.672 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.669+0000 7fca5fa10180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T21:57:20.705 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.701+0000 7f0664615640 1 Processor -- start 2023-12-16T21:57:20.705 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.701+0000 7f0664615640 1 -- start start 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.701+0000 7f0664615640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f065c15ba60 0x7f065c071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.701+0000 7f0664615640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c06a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.701+0000 7f0664615640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065c0724a0 con 0x7f065c15be40 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.701+0000 7f0664615640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c071c00 con 0x7f065c15ba60 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.701+0000 7f0664615640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c071ee0 con 0x7f065c15c2a0 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f0662bb2640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c06a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f06623b1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f065c15ba60 0x7f065c071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f06623b1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f065c15ba60 0x7f065c071540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40446/0 (socket says 172.21.15.122:40446) 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f06623b1640 1 -- 172.21.15.122:0/242041297 learned_addr learned my addr 172.21.15.122:0/242041297 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f0662bb2640 1 -- 172.21.15.122:0/242041297 >> v2:172.21.15.122:3300/0 conn(0x7f065c15ba60 msgr2=0x7f065c071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f0662bb2640 1 --2- 172.21.15.122:0/242041297 >> v2:172.21.15.122:3300/0 conn(0x7f065c15ba60 0x7f065c071540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f0662bb2640 1 -- 172.21.15.122:0/242041297 >> v1:172.21.15.28:6789/0 conn(0x7f065c15be40 legacy=0x7f065c06ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.707 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f0662bb2640 1 -- 172.21.15.122:0/242041297 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c0721c0 con 0x7f065c15c2a0 2023-12-16T21:57:20.708 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f06623b1640 1 --2- 172.21.15.122:0/242041297 >> v2:172.21.15.122:3300/0 conn(0x7f065c15ba60 0x7f065c071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:20.710 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f0662bb2640 1 --2- 172.21.15.122:0/242041297 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c06a530 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0654009d90 tx=0x7f065401c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66914640fcf3433e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:20.710 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f06613af640 1 -- 172.21.15.122:0/242041297 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540a5860 con 0x7f065c15c2a0 2023-12-16T21:57:20.710 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f06613af640 1 -- 172.21.15.122:0/242041297 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06540a59c0 con 0x7f065c15c2a0 2023-12-16T21:57:20.710 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.705+0000 7f06613af640 1 -- 172.21.15.122:0/242041297 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540a5d50 con 0x7f065c15c2a0 2023-12-16T21:57:20.711 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- 172.21.15.122:0/242041297 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 msgr2=0x7f065c06a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.711 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 --2- 172.21.15.122:0/242041297 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c06a530 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0654009d90 tx=0x7f065401c890 comp rx=0 tx=0).stop 2023-12-16T21:57:20.711 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- 172.21.15.122:0/242041297 shutdown_connections 2023-12-16T21:57:20.711 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 --2- 172.21.15.122:0/242041297 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c06a530 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.711 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 --2- 172.21.15.122:0/242041297 >> v2:172.21.15.122:3300/0 conn(0x7f065c15ba60 0x7f065c071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.711 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- 172.21.15.122:0/242041297 >> 172.21.15.122:0/242041297 conn(0x7f065c0b1db0 msgr2=0x7f065c0b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:20.712 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- 172.21.15.122:0/242041297 shutdown_connections 2023-12-16T21:57:20.712 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- 172.21.15.122:0/242041297 wait complete. 2023-12-16T21:57:20.712 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 Processor -- start 2023-12-16T21:57:20.712 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- start start 2023-12-16T21:57:20.714 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c15ba60 0x7f065c155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.714 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f065c15be40 0x7f065c134610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c134b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c146440 con 0x7f065c15be40 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1509f0 con 0x7f065c15ba60 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c150cd0 con 0x7f065c15c2a0 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c15ba60 0x7f065c155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0662bb2640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c134b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c15ba60 0x7f065c155c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40452/0 (socket says 172.21.15.122:40452) 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0662bb2640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c134b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.122:50128/0 (socket says 172.21.15.122:50128) 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 -- 172.21.15.122:0/1488159115 learned_addr learned my addr 172.21.15.122:0/1488159115 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0661bb0640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f065c15be40 0x7f065c134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 -- 172.21.15.122:0/1488159115 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 msgr2=0x7f065c134b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c134b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.715 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 -- 172.21.15.122:0/1488159115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f065c15be40 msgr2=0x7f065c134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f065c15be40 0x7f065c134610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 -- 172.21.15.122:0/1488159115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c0094d0 con 0x7f065c15ba60 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0661bb0640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f065c15be40 0x7f065c134610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f06623b1640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c15ba60 0x7f065c155c20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f064c00e9d0 tx=0x7f064c00ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0652ffd640 1 -- 172.21.15.122:0/1488159115 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c00ca60 con 0x7f065c15ba60 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0652ffd640 1 -- 172.21.15.122:0/1488159115 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f064c00cbc0 con 0x7f065c15ba60 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0654009a70 con 0x7f065c15ba60 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0652ffd640 1 -- 172.21.15.122:0/1488159115 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c013470 con 0x7f065c15ba60 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c151290 con 0x7f065c15ba60 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0652ffd640 1 -- 172.21.15.122:0/1488159115 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f064c005ca0 con 0x7f065c15ba60 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.709+0000 7f0652ffd640 1 -- 172.21.15.122:0/1488159115 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f064c013bc0 con 0x7f065c15ba60 2023-12-16T21:57:20.716 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.713+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c0674d0 con 0x7f065c15ba60 2023-12-16T21:57:20.718 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.713+0000 7f0652ffd640 1 -- 172.21.15.122:0/1488159115 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f064c021060 con 0x7f065c15ba60 2023-12-16T21:57:20.787 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.785+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b7d9aa20-7c15-425a-922b-659558ce1ef0", "id": 6} v 0) v1 -- 0x7f065c06b330 con 0x7f065c15ba60 2023-12-16T21:57:20.795 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0652ffd640 1 -- 172.21.15.122:0/1488159115 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7d9aa20-7c15-425a-922b-659558ce1ef0", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f064c017070 con 0x7f065c15ba60 2023-12-16T21:57:20.855 INFO:teuthology.orchestra.run.smithi122.stdout:6 2023-12-16T21:57:20.856 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c15ba60 msgr2=0x7f065c155c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:20.856 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c15ba60 0x7f065c155c20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f064c00e9d0 tx=0x7f064c00ee90 comp rx=0 tx=0).stop 2023-12-16T21:57:20.856 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 shutdown_connections 2023-12-16T21:57:20.856 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f065c15c2a0 0x7f065c134b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.856 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f065c15be40 0x7f065c134610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.857 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 --2- 172.21.15.122:0/1488159115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f065c15ba60 0x7f065c155c20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:20.857 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 >> 172.21.15.122:0/1488159115 conn(0x7f065c0b1db0 msgr2=0x7f065c1532e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:20.857 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 shutdown_connections 2023-12-16T21:57:20.857 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.793+0000 7f0664615640 1 -- 172.21.15.122:0/1488159115 wait complete. 2023-12-16T21:57:20.857 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd new 3160ca90-072b-47b7-9f5a-e9a306cdf4d3 7 2023-12-16T21:57:20.857 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.821+0000 7fca5fa10180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T21:57:20.858 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.798+0000 7ff19134b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T21:57:20.860 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:20.858+0000 7ff19134b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T21:57:20.885 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:20.881+0000 7fca5fa10180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T21:57:21.001 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0f532d640 1 Processor -- start 2023-12-16T21:57:21.001 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0f532d640 1 -- start start 2023-12-16T21:57:21.002 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0f532d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc0f01449d0 0x7fc0f006a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.002 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0f532d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f00cc6a0 con 0x7fc0f0071590 2023-12-16T21:57:21.002 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0f532d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f0144db0 con 0x7fc0f015f5b0 2023-12-16T21:57:21.002 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0f532d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f006ac30 con 0x7fc0f01449d0 2023-12-16T21:57:21.002 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0eeffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc0f01449d0 0x7fc0f006a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.002 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0eeffd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc0f01449d0 0x7fc0f006a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40462/0 (socket says 172.21.15.122:40462) 2023-12-16T21:57:21.002 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0eeffd640 1 -- 172.21.15.122:0/971563849 learned_addr learned my addr 172.21.15.122:0/971563849 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:21.002 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0ee7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc0f015f5b0 0x7fc0f00cd0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58590/0 (socket says 172.21.15.122:58590) 2023-12-16T21:57:21.003 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:20.997+0000 7fc0eeffd640 1 -- 172.21.15.122:0/971563849 >> v1:172.21.15.122:6789/0 conn(0x7fc0f015f5b0 legacy=0x7fc0f00cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.003 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0eeffd640 1 -- 172.21.15.122:0/971563849 >> v1:172.21.15.28:6789/0 conn(0x7fc0f0071590 legacy=0x7fc0f00cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:21.003 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0eeffd640 1 -- 172.21.15.122:0/971563849 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f006af10 con 0x7fc0f01449d0 2023-12-16T21:57:21.004 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0eeffd640 1 --2- 172.21.15.122:0/971563849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f01449d0 0x7fc0f006a530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4009870 tx=0x7fc0e401c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bad2b6b5546a7dca server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:21.004 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0edffb640 1 -- 172.21.15.122:0/971563849 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40aa660 con 0x7fc0f01449d0 2023-12-16T21:57:21.005 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0edffb640 1 -- 172.21.15.122:0/971563849 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e40aa7c0 con 0x7fc0f01449d0 2023-12-16T21:57:21.005 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0edffb640 1 -- 172.21.15.122:0/971563849 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40aab50 con 0x7fc0f01449d0 2023-12-16T21:57:21.006 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0f532d640 1 -- 172.21.15.122:0/971563849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f01449d0 msgr2=0x7fc0f006a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.006 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0f532d640 1 --2- 172.21.15.122:0/971563849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f01449d0 0x7fc0f006a530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4009870 tx=0x7fc0e401c500 comp rx=0 tx=0).stop 2023-12-16T21:57:21.006 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0f532d640 1 -- 172.21.15.122:0/971563849 shutdown_connections 2023-12-16T21:57:21.006 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0f532d640 1 --2- 172.21.15.122:0/971563849 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f01449d0 0x7fc0f006a530 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.006 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0f532d640 1 -- 172.21.15.122:0/971563849 >> 172.21.15.122:0/971563849 conn(0x7fc0f006d510 msgr2=0x7fc0f006d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:21.006 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0f532d640 1 -- 172.21.15.122:0/971563849 shutdown_connections 2023-12-16T21:57:21.006 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.001+0000 7fc0f532d640 1 -- 172.21.15.122:0/971563849 wait complete. 2023-12-16T21:57:21.007 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 Processor -- start 2023-12-16T21:57:21.007 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 -- start start 2023-12-16T21:57:21.009 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f0071590 0x7fc0f0134550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.009 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc0f01449d0 0x7fc0f0139720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.009 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f015f5b0 0x7fc0f01f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.009 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01f4120 con 0x7fc0f015f5b0 2023-12-16T21:57:21.009 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0134ec0 con 0x7fc0f0071590 2023-12-16T21:57:21.009 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f01351a0 con 0x7fc0f01449d0 2023-12-16T21:57:21.009 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f0071590 0x7fc0f0134550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.009 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0ee7fc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc0f01449d0 0x7fc0f0139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f0071590 0x7fc0f0134550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40478/0 (socket says 172.21.15.122:40478) 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0ee7fc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc0f01449d0 0x7fc0f0139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.122:50142/0 (socket says 172.21.15.122:50142) 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 -- 172.21.15.122:0/3533643478 learned_addr learned my addr 172.21.15.122:0/3533643478 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0ef7fe640 1 --2- 172.21.15.122:0/3533643478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f015f5b0 0x7fc0f01f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 -- 172.21.15.122:0/3533643478 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc0f01449d0 msgr2=0x7fc0f0139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 --2- 172.21.15.122:0/3533643478 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc0f01449d0 0x7fc0f0139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 -- 172.21.15.122:0/3533643478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f015f5b0 msgr2=0x7fc0f01f45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 --2- 172.21.15.122:0/3533643478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f015f5b0 0x7fc0f01f45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 -- 172.21.15.122:0/3533643478 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e40094d0 con 0x7fc0f0071590 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0eeffd640 1 --2- 172.21.15.122:0/3533643478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f0071590 0x7fc0f0134550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40ac040 tx=0x7fc0e401ad40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0d77fe640 1 -- 172.21.15.122:0/3533643478 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40b38c0 con 0x7fc0f0071590 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0d77fe640 1 -- 172.21.15.122:0/3533643478 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e40b3a40 con 0x7fc0f0071590 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f0135480 con 0x7fc0f0071590 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0d77fe640 1 -- 172.21.15.122:0/3533643478 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e40b3df0 con 0x7fc0f0071590 2023-12-16T21:57:21.010 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f0134a90 con 0x7fc0f0071590 2023-12-16T21:57:21.011 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0d77fe640 1 -- 172.21.15.122:0/3533643478 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc0e40bd6f0 con 0x7fc0f0071590 2023-12-16T21:57:21.011 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0d77fe640 1 -- 172.21.15.122:0/3533643478 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fc0e40a6040 con 0x7fc0f0071590 2023-12-16T21:57:21.011 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.005+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f00674d0 con 0x7fc0f0071590 2023-12-16T21:57:21.011 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.009+0000 7fc0d77fe640 1 -- 172.21.15.122:0/3533643478 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7fc0e40c3060 con 0x7fc0f0071590 2023-12-16T21:57:21.078 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.077+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3160ca90-072b-47b7-9f5a-e9a306cdf4d3", "id": 7} v 0) v1 -- 0x7fc0f01f3ef0 con 0x7fc0f0071590 2023-12-16T21:57:21.086 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.081+0000 7fc0d77fe640 1 -- 172.21.15.122:0/3533643478 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3160ca90-072b-47b7-9f5a-e9a306cdf4d3", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc0e40b2d00 con 0x7fc0f0071590 2023-12-16T21:57:21.086 INFO:teuthology.orchestra.run.smithi122.stdout:7 2023-12-16T21:57:21.087 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f0071590 msgr2=0x7fc0f0134550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.088 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 --2- 172.21.15.122:0/3533643478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f0071590 0x7fc0f0134550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40ac040 tx=0x7fc0e401ad40 comp rx=0 tx=0).stop 2023-12-16T21:57:21.088 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 shutdown_connections 2023-12-16T21:57:21.088 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 --2- 172.21.15.122:0/3533643478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f015f5b0 0x7fc0f01f45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.088 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 --2- 172.21.15.122:0/3533643478 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc0f01449d0 0x7fc0f0139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.088 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 --2- 172.21.15.122:0/3533643478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0f0071590 0x7fc0f0134550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.088 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 >> 172.21.15.122:0/3533643478 conn(0x7fc0f006d510 msgr2=0x7fc0f015e000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:21.088 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 shutdown_connections 2023-12-16T21:57:21.088 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:57:21.085+0000 7fc0f532d640 1 -- 172.21.15.122:0/3533643478 wait complete. 2023-12-16T21:57:21.103 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-16T21:57:21.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-16T21:57:21.106 INFO:tasks.ceph.osd.0:Started 2023-12-16T21:57:21.106 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-16T21:57:21.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-16T21:57:21.109 INFO:tasks.ceph.osd.1:Started 2023-12-16T21:57:21.109 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-16T21:57:21.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-16T21:57:21.111 INFO:tasks.ceph.osd.2:Started 2023-12-16T21:57:21.112 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-16T21:57:21.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-16T21:57:21.114 INFO:tasks.ceph.osd.3:Started 2023-12-16T21:57:21.114 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-16T21:57:21.114 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-16T21:57:21.117 INFO:tasks.ceph.osd.4:Started 2023-12-16T21:57:21.117 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-16T21:57:21.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-16T21:57:21.119 INFO:tasks.ceph.osd.5:Started 2023-12-16T21:57:21.119 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-16T21:57:21.119 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-12-16T21:57:21.122 INFO:tasks.ceph.osd.6:Started 2023-12-16T21:57:21.122 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-16T21:57:21.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-12-16T21:57:21.124 INFO:tasks.ceph.osd.7:Started 2023-12-16T21:57:21.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40be7d640 1 Processor -- start 2023-12-16T21:57:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40be7d640 1 -- start start 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40be7d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc4041557d0 0x7fc40415ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40be7d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4040b41f0 con 0x7fc4040b4c40 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40be7d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40415f670 con 0x7fc40415f110 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40be7d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40415f950 con 0x7fc4041557d0 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40a41a640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc4040b4c40 0x7fc4040d1420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39720/0 (socket says 172.21.15.28:39720) 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40a41a640 1 -- 172.21.15.28:0/211024533 learned_addr learned my addr 172.21.15.28:0/211024533 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905538903 0 0) 0x7fc4040b41f0 con 0x7fc4040b4c40 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f0003530 con 0x7fc4040b4c40 2023-12-16T21:57:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc409c19640 1 --2- 172.21.15.28:0/211024533 >> v2:172.21.15.122:3300/0 conn(0x7fc4041557d0 0x7fc40415ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2550229822 0 0) 0x7fc3f0003530 con 0x7fc4040b4c40 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 >> v2:172.21.15.122:3300/0 conn(0x7fc4041557d0 msgr2=0x7fc40415ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 --2- 172.21.15.28:0/211024533 >> v2:172.21.15.122:3300/0 conn(0x7fc4041557d0 0x7fc40415ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 >> v1:172.21.15.122:6789/0 conn(0x7fc40415f110 legacy=0x7fc4040cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4040b4440 con 0x7fc4040b4c40 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fc3fc001a00 con 0x7fc4040b4c40 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 >> v1:172.21.15.28:6789/0 conn(0x7fc4040b4c40 legacy=0x7fc4040d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f0003980 0x7fc3f0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc3f00047d0 0x7fc3f0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0005190 0x7fc3f000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40415f950 con 0x7fc3f0003980 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f000c130 con 0x7fc3f0005190 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f000c2f0 con 0x7fc3f00047d0 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc409c19640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f0003980 0x7fc3f0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40a41a640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0005190 0x7fc3f000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc409418640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc3f00047d0 0x7fc3f0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc409c19640 1 -- 172.21.15.28:0/211024533 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc3f00047d0 msgr2=0x7fc3f0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc409c19640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc3f00047d0 0x7fc3f0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc409c19640 1 -- 172.21.15.28:0/211024533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0005190 msgr2=0x7fc3f000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc409c19640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0005190 0x7fc3f000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc40a41a640 1 -- 172.21.15.28:0/211024533 reap_dead start 2023-12-16T21:57:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.278+0000 7fc409c19640 1 -- 172.21.15.28:0/211024533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f0003530 con 0x7fc3f0003980 2023-12-16T21:57:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409c19640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f0003980 0x7fc3f0003da0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc400009ae0 tx=0x7fc40001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebf6ab976f54b9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4000a6510 con 0x7fc3f0003980 2023-12-16T21:57:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4000a6670 con 0x7fc3f0003980 2023-12-16T21:57:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc408c17640 1 -- 172.21.15.28:0/211024533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4000a6a20 con 0x7fc3f0003980 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- 172.21.15.28:0/211024533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f0003980 msgr2=0x7fc3f0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f0003980 0x7fc3f0003da0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc400009ae0 tx=0x7fc40001ca20 comp rx=0 tx=0).stop 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- 172.21.15.28:0/211024533 shutdown_connections 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 --2- 172.21.15.28:0/211024533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f0003980 0x7fc3f0003da0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- 172.21.15.28:0/211024533 >> 172.21.15.28:0/211024533 conn(0x7fc40406a930 msgr2=0x7fc40406ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- 172.21.15.28:0/211024533 shutdown_connections 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- 172.21.15.28:0/211024533 wait complete. 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 Processor -- start 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- start start 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0003980 0x7fc404156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc40414d1e0 0x7fc40413b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc40414d660 0x7fc40413bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4041396e0 con 0x7fc40414d1e0 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc404157570 con 0x7fc3f0003980 2023-12-16T21:57:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc404157850 con 0x7fc40414d660 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc40414d1e0 0x7fc40413b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc40414d1e0 0x7fc40413b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40930/0 (socket says 172.21.15.28:40930) 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 -- 172.21.15.28:0/1583362851 learned_addr learned my addr 172.21.15.28:0/1583362851 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409c19640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0003980 0x7fc404156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40a41a640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc40414d660 0x7fc40413bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 -- 172.21.15.28:0/1583362851 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc40414d660 msgr2=0x7fc40413bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc40414d660 0x7fc40413bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 -- 172.21.15.28:0/1583362851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0003980 msgr2=0x7fc404156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0003980 0x7fc404156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 -- 172.21.15.28:0/1583362851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc404157b30 con 0x7fc40414d1e0 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc409418640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc40414d1e0 0x7fc40413b9a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400f5f0 tx=0x7fc3f400f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc3fa7fc640 1 -- 172.21.15.28:0/1583362851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40052e0 con 0x7fc40414d1e0 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc3fa7fc640 1 -- 172.21.15.28:0/1583362851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f4005440 con 0x7fc40414d1e0 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc404157e10 con 0x7fc40414d1e0 2023-12-16T21:57:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc3fa7fc640 1 -- 172.21.15.28:0/1583362851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4003820 con 0x7fc40414d1e0 2023-12-16T21:57:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.282+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc404158610 con 0x7fc40414d1e0 2023-12-16T21:57:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.286+0000 7fc3fa7fc640 1 -- 172.21.15.28:0/1583362851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc3f4003c80 con 0x7fc40414d1e0 2023-12-16T21:57:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.286+0000 7fc3fa7fc640 1 -- 172.21.15.28:0/1583362851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc3f401dd20 con 0x7fc40414d1e0 2023-12-16T21:57:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.286+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4040674d0 con 0x7fc40414d1e0 2023-12-16T21:57:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.286+0000 7fc3fa7fc640 1 -- 172.21.15.28:0/1583362851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7fc3f401d460 con 0x7fc40414d1e0 2023-12-16T21:57:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.342+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc404070ca0 con 0x7fc40414d1e0 2023-12-16T21:57:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.342+0000 7fc3fa7fc640 1 -- 172.21.15.28:0/1583362851 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc3f402c090 con 0x7fc40414d1e0 2023-12-16T21:57:21.345 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:21.345 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:21.083225+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc40414d1e0 msgr2=0x7fc40413b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc40414d1e0 0x7fc40413b9a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400f5f0 tx=0x7fc3f400f9f0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 shutdown_connections 2023-12-16T21:57:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc40414d660 0x7fc40413bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc40414d1e0 0x7fc40413b9a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 --2- 172.21.15.28:0/1583362851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3f0003980 0x7fc404156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 >> 172.21.15.28:0/1583362851 conn(0x7fc40406a930 msgr2=0x7fc40406ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 shutdown_connections 2023-12-16T21:57:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:21.346+0000 7fc40be7d640 1 -- 172.21.15.28:0/1583362851 wait complete. 2023-12-16T21:57:21.360 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-16T21:57:21.360 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-16T21:57:21.494 INFO:tasks.ceph.mgr.x.smithi122.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-16T21:57:21.494 INFO:tasks.ceph.mgr.x.smithi122.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-16T21:57:21.494 INFO:tasks.ceph.mgr.x.smithi122.stderr: from numpy import show_config as show_numpy_config 2023-12-16T21:57:21.495 INFO:tasks.ceph.mgr.y.smithi028.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-16T21:57:21.495 INFO:tasks.ceph.mgr.y.smithi028.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-16T21:57:21.495 INFO:tasks.ceph.mgr.y.smithi028.stderr: from numpy import show_config as show_numpy_config 2023-12-16T21:57:21.512 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-16T21:57:21.509+0000 7f0e053b8640 -1 Falling back to public interface 2023-12-16T21:57:21.534 INFO:tasks.ceph.osd.3.smithi028.stderr:2023-12-16T21:57:21.534+0000 7fa1be762640 -1 Falling back to public interface 2023-12-16T21:57:21.535 INFO:tasks.ceph.osd.2.smithi028.stderr:2023-12-16T21:57:21.534+0000 7fbf71a31640 -1 Falling back to public interface 2023-12-16T21:57:21.535 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-16T21:57:21.534+0000 7f4d3b193640 -1 Falling back to public interface 2023-12-16T21:57:21.536 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-16T21:57:21.533+0000 7f703079b640 -1 Falling back to public interface 2023-12-16T21:57:21.538 INFO:tasks.ceph.osd.1.smithi028.stderr:2023-12-16T21:57:21.538+0000 7f56e485c640 -1 Falling back to public interface 2023-12-16T21:57:21.547 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:21.545+0000 7fca5fa10180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T21:57:21.547 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:21.546+0000 7ff19134b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T21:57:21.679 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:21.678+0000 7ff19134b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T21:57:21.689 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:21.685+0000 7fca5fa10180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T21:57:21.760 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-16T21:57:21.757+0000 7f669ad2d640 -1 Falling back to public interface 2023-12-16T21:57:21.787 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-16T21:57:21.785+0000 7f5afb432640 -1 Falling back to public interface 2023-12-16T21:57:21.840 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:21.838+0000 7ff19134b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T21:57:21.873 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:21.869+0000 7fca5fa10180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T21:57:21.905 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:21.906+0000 7ff19134b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T21:57:21.943 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:21.941+0000 7fca5fa10180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T21:57:22.034 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:22.034+0000 7ff19134b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T21:57:22.077 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:22.073+0000 7fca5fa10180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T21:57:22.165 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:22.272 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.270+0000 7f9c341a3640 1 Processor -- start 2023-12-16T21:57:22.273 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.270+0000 7f9c341a3640 1 -- start start 2023-12-16T21:57:22.274 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 0x7f9c2c0cd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:22.274 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c2c0b6a60 0x7f9c2c0cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9c2c15ecf0 0x7f9c2c15f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c15f6a0 con 0x7f9c2c0b6a60 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c0cb700 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c0d0720 con 0x7f9c2c0b6680 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c2c0b6a60 0x7f9c2c0cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c2c0b6a60 0x7f9c2c0cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40946/0 (socket says 172.21.15.28:40946) 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 -- 172.21.15.28:0/3496301299 learned_addr learned my addr 172.21.15.28:0/3496301299 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c31f3f640 1 --2- 172.21.15.28:0/3496301299 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 0x7f9c2c0cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c32740640 1 --2- 172.21.15.28:0/3496301299 >> v2:172.21.15.122:3300/0 conn(0x7f9c2c15ecf0 0x7f9c2c15f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 -- 172.21.15.28:0/3496301299 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 msgr2=0x7f9c2c0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 --2- 172.21.15.28:0/3496301299 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 0x7f9c2c0cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 -- 172.21.15.28:0/3496301299 >> v2:172.21.15.122:3300/0 conn(0x7f9c2c15ecf0 msgr2=0x7f9c2c15f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 --2- 172.21.15.28:0/3496301299 >> v2:172.21.15.122:3300/0 conn(0x7f9c2c15ecf0 0x7f9c2c15f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.275 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 -- 172.21.15.28:0/3496301299 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c0cb9a0 con 0x7f9c2c0b6a60 2023-12-16T21:57:22.276 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c3173e640 1 --2- 172.21.15.28:0/3496301299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c0b6a60 0x7f9c2c0cf920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009870 tx=0x7f9c1c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beebcccc59f8a9a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:22.276 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c30f3d640 1 -- 172.21.15.28:0/3496301299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0ab5f0 con 0x7f9c2c0b6a60 2023-12-16T21:57:22.276 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c30f3d640 1 -- 172.21.15.28:0/3496301299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c0ab750 con 0x7f9c2c0b6a60 2023-12-16T21:57:22.276 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c30f3d640 1 -- 172.21.15.28:0/3496301299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0abae0 con 0x7f9c2c0b6a60 2023-12-16T21:57:22.277 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 -- 172.21.15.28:0/3496301299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c0b6a60 msgr2=0x7f9c2c0cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:22.277 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 --2- 172.21.15.28:0/3496301299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c0b6a60 0x7f9c2c0cf920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009870 tx=0x7f9c1c01c4d0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.277 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 -- 172.21.15.28:0/3496301299 shutdown_connections 2023-12-16T21:57:22.277 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 --2- 172.21.15.28:0/3496301299 >> v2:172.21.15.122:3300/0 conn(0x7f9c2c15ecf0 0x7f9c2c15f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.277 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 --2- 172.21.15.28:0/3496301299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c0b6a60 0x7f9c2c0cf920 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.278 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 --2- 172.21.15.28:0/3496301299 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 0x7f9c2c0cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.278 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 -- 172.21.15.28:0/3496301299 >> 172.21.15.28:0/3496301299 conn(0x7f9c2c06e910 msgr2=0x7f9c2c06ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:22.278 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.274+0000 7f9c341a3640 1 -- 172.21.15.28:0/3496301299 shutdown_connections 2023-12-16T21:57:22.278 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 -- 172.21.15.28:0/3496301299 wait complete. 2023-12-16T21:57:22.278 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 Processor -- start 2023-12-16T21:57:22.278 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 -- start start 2023-12-16T21:57:22.280 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 0x7f9c2c14aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:22.280 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c2c0b6a60 0x7f9c2c14a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c15ecf0 0x7f9c2c0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c06ced0 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c06d1b0 con 0x7f9c2c0b6a60 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c06d490 con 0x7f9c2c0b6680 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c15ecf0 0x7f9c2c0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c15ecf0 0x7f9c2c0cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40954/0 (socket says 172.21.15.28:40954) 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 -- 172.21.15.28:0/3075460842 learned_addr learned my addr 172.21.15.28:0/3075460842 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c3173e640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c2c0b6a60 0x7f9c2c14a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c31f3f640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 0x7f9c2c14aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 -- 172.21.15.28:0/3075460842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 msgr2=0x7f9c2c14aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 0x7f9c2c14aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 -- 172.21.15.28:0/3075460842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c2c0b6a60 msgr2=0x7f9c2c14a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:22.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c2c0b6a60 0x7f9c2c14a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 -- 172.21.15.28:0/3075460842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c0094d0 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c32740640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c15ecf0 0x7f9c2c0cace0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9c24007e40 tx=0x7f9c2400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c227fc640 1 -- 172.21.15.28:0/3075460842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24014620 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c227fc640 1 -- 172.21.15.28:0/3075460842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2400ee90 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c0ca4e0 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c227fc640 1 -- 172.21.15.28:0/3075460842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c240149e0 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c14a670 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c227fc640 1 -- 172.21.15.28:0/3075460842 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9c2401e460 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c227fc640 1 -- 172.21.15.28:0/3075460842 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9c2401da90 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.278+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c0ca340 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.283 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.282+0000 7f9c227fc640 1 -- 172.21.15.28:0/3075460842 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f9c2c0ca340 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.339 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.338+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c2c0674d0 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.340 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.338+0000 7f9c227fc640 1 -- 172.21.15.28:0/3075460842 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9c2401d3c0 con 0x7f9c2c15ecf0 2023-12-16T21:57:22.340 INFO:teuthology.misc.health.smithi028.stdout: 2023-12-16T21:57:22.340 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":10,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:21.083225+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:22.341 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c15ecf0 msgr2=0x7f9c2c0cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:22.342 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c15ecf0 0x7f9c2c0cace0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9c24007e40 tx=0x7f9c2400e4b0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.342 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 shutdown_connections 2023-12-16T21:57:22.342 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c2c15ecf0 0x7f9c2c0cace0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.342 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c2c0b6a60 0x7f9c2c14a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.342 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 --2- 172.21.15.28:0/3075460842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9c2c0b6680 0x7f9c2c14aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:22.342 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 >> 172.21.15.28:0/3075460842 conn(0x7f9c2c06e910 msgr2=0x7f9c2c15e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:22.342 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 shutdown_connections 2023-12-16T21:57:22.342 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:22.342+0000 7f9c341a3640 1 -- 172.21.15.28:0/3075460842 wait complete. 2023-12-16T21:57:22.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-16T21:57:22.657 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:22.658+0000 7ff19134b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T21:57:22.675 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:22.673+0000 7fca5fa10180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T21:57:22.736 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:22.734+0000 7ff19134b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T21:57:22.751 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:22.749+0000 7fca5fa10180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T21:57:22.915 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:22.914+0000 7ff19134b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T21:57:22.923 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:22.921+0000 7fca5fa10180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T21:57:22.991 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:22.990+0000 7ff19134b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T21:57:22.996 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:22.993+0000 7fca5fa10180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T21:57:23.286 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:23.281+0000 7fca5fa10180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T21:57:23.293 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:23.290+0000 7ff19134b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T21:57:23.345 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:23.341+0000 7fca5fa10180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T21:57:23.354 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-16T21:57:23.354+0000 7ff19134b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T21:57:24.391 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:24.389+0000 7fca07199640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-16T21:57:24.391 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-16T21:57:24.389+0000 7fca07199640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-16T21:57:25.367 INFO:tasks.ceph.osd.2.smithi028.stderr:2023-12-16T21:57:25.366+0000 7fbf71a31640 -1 osd.2 0 log_to_monitors true 2023-12-16T21:57:25.424 INFO:tasks.ceph.osd.3.smithi028.stderr:2023-12-16T21:57:25.422+0000 7fa1be762640 -1 osd.3 0 log_to_monitors true 2023-12-16T21:57:25.430 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-16T21:57:25.425+0000 7f703079b640 -1 osd.7 0 log_to_monitors true 2023-12-16T21:57:25.438 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-16T21:57:25.438+0000 7f4d3b193640 -1 osd.0 0 log_to_monitors true 2023-12-16T21:57:25.449 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-16T21:57:25.445+0000 7f0e053b8640 -1 osd.4 0 log_to_monitors true 2023-12-16T21:57:25.458 INFO:tasks.ceph.osd.1.smithi028.stderr:2023-12-16T21:57:25.458+0000 7f56e485c640 -1 osd.1 0 log_to_monitors true 2023-12-16T21:57:25.590 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-16T21:57:25.585+0000 7f5afb432640 -1 osd.6 0 log_to_monitors true 2023-12-16T21:57:25.667 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-16T21:57:25.665+0000 7f669ad2d640 -1 osd.5 0 log_to_monitors true 2023-12-16T21:57:27.397 INFO:tasks.ceph.osd.2.smithi028.stderr:2023-12-16T21:57:27.394+0000 7fbf6d1b7640 -1 osd.2 0 waiting for initial osdmap 2023-12-16T21:57:27.426 INFO:tasks.ceph.osd.2.smithi028.stderr:2023-12-16T21:57:27.426+0000 7fbf65804640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:57:28.701 INFO:tasks.ceph.osd.1.smithi028.stderr:2023-12-16T21:57:28.697+0000 7f56dffe2640 -1 osd.1 0 waiting for initial osdmap 2023-12-16T21:57:28.722 INFO:tasks.ceph.osd.1.smithi028.stderr:2023-12-16T21:57:28.721+0000 7f56d862f640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:57:28.730 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-16T21:57:28.729+0000 7f4d36919640 -1 osd.0 0 waiting for initial osdmap 2023-12-16T21:57:28.742 INFO:tasks.ceph.osd.3.smithi028.stderr:2023-12-16T21:57:28.741+0000 7fa1b9ee8640 -1 osd.3 0 waiting for initial osdmap 2023-12-16T21:57:28.762 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-16T21:57:28.761+0000 7f4d2ef66640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:57:28.774 INFO:tasks.ceph.osd.3.smithi028.stderr:2023-12-16T21:57:28.773+0000 7fa1b2535640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:57:29.162 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:29.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.277+0000 7fd87df02640 1 Processor -- start 2023-12-16T21:57:29.281 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.277+0000 7fd87df02640 1 -- start start 2023-12-16T21:57:29.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd87813dce0 0x7fd87813faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:29.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd878070a00 con 0x7fd87813cfa0 2023-12-16T21:57:29.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8781401d0 con 0x7fd87813f630 2023-12-16T21:57:29.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87813d380 con 0x7fd87813dce0 2023-12-16T21:57:29.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd877fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd87813dce0 0x7fd87813faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:29.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd877fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd87813dce0 0x7fd87813faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40368/0 (socket says 172.21.15.28:40368) 2023-12-16T21:57:29.282 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd877fff640 1 -- 172.21.15.28:0/4016558611 learned_addr learned my addr 172.21.15.28:0/4016558611 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:29.283 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd877fff640 1 -- 172.21.15.28:0/4016558611 >> v1:172.21.15.122:6789/0 conn(0x7fd87813f630 legacy=0x7fd87813d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:29.283 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd877fff640 1 -- 172.21.15.28:0/4016558611 >> v1:172.21.15.28:6789/0 conn(0x7fd87813cfa0 legacy=0x7fd8780ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:29.283 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd877fff640 1 -- 172.21.15.28:0/4016558611 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8781404b0 con 0x7fd87813dce0 2023-12-16T21:57:29.284 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd877fff640 1 --2- 172.21.15.28:0/4016558611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813dce0 0x7fd87813faa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd870009e50 tx=0x7fd87001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e9f97b2e4f1b11 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:29.285 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd8767fc640 1 -- 172.21.15.28:0/4016558611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8700a5730 con 0x7fd87813dce0 2023-12-16T21:57:29.285 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd8767fc640 1 -- 172.21.15.28:0/4016558611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8700a5890 con 0x7fd87813dce0 2023-12-16T21:57:29.285 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd8767fc640 1 -- 172.21.15.28:0/4016558611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8700a5c20 con 0x7fd87813dce0 2023-12-16T21:57:29.285 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 -- 172.21.15.28:0/4016558611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813dce0 msgr2=0x7fd87813faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:29.286 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 --2- 172.21.15.28:0/4016558611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813dce0 0x7fd87813faa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd870009e50 tx=0x7fd87001c980 comp rx=0 tx=0).stop 2023-12-16T21:57:29.286 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 -- 172.21.15.28:0/4016558611 shutdown_connections 2023-12-16T21:57:29.286 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 --2- 172.21.15.28:0/4016558611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813dce0 0x7fd87813faa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:29.286 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 -- 172.21.15.28:0/4016558611 >> 172.21.15.28:0/4016558611 conn(0x7fd87806c5e0 msgr2=0x7fd87806c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:29.286 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.281+0000 7fd87df02640 1 -- 172.21.15.28:0/4016558611 shutdown_connections 2023-12-16T21:57:29.286 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 -- 172.21.15.28:0/4016558611 wait complete. 2023-12-16T21:57:29.288 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 Processor -- start 2023-12-16T21:57:29.288 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 -- start start 2023-12-16T21:57:29.291 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd87813cfa0 0x7fd8780afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:29.291 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813f630 0x7fd878154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:29.291 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8780b1540 0x7fd878136410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:29.291 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd878136950 con 0x7fd87813f630 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87815d5d0 con 0x7fd8780b1540 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8780b04c0 con 0x7fd87813cfa0 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813f630 0x7fd878154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813f630 0x7fd878154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40378/0 (socket says 172.21.15.28:40378) 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 -- 172.21.15.28:0/1348165143 learned_addr learned my addr 172.21.15.28:0/1348165143 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd877fff640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8780b1540 0x7fd878136410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd8777fe640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd87813cfa0 0x7fd8780afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 -- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd87813cfa0 msgr2=0x7fd8780afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd87813cfa0 0x7fd8780afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 -- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8780b1540 msgr2=0x7fd878136410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:29.292 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8780b1540 0x7fd878136410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 -- 172.21.15.28:0/1348165143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd870009b30 con 0x7fd87813f630 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd8777fe640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd87813cfa0 0x7fd8780afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd876ffd640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813f630 0x7fd878154300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00c3c0 tx=0x7fd86c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd85ffff640 1 -- 172.21.15.28:0/1348165143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c0052d0 con 0x7fd87813f630 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd85ffff640 1 -- 172.21.15.28:0/1348165143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c005430 con 0x7fd87813f630 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8780b0780 con 0x7fd87813f630 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd85ffff640 1 -- 172.21.15.28:0/1348165143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c003840 con 0x7fd87813f630 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8780b0d40 con 0x7fd87813f630 2023-12-16T21:57:29.293 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.285+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8780694b0 con 0x7fd87813f630 2023-12-16T21:57:29.294 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.289+0000 7fd85ffff640 1 -- 172.21.15.28:0/1348165143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 85798+0+0 (secure 0 0 0) 0x7fd86c0039a0 con 0x7fd87813f630 2023-12-16T21:57:29.294 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.289+0000 7fd85ffff640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd864066410 0x7fd8640688d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:29.294 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.289+0000 7fd85ffff640 1 -- 172.21.15.28:0/1348165143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2511+0+0 (secure 0 0 0) 0x7fd86c0825d0 con 0x7fd87813f630 2023-12-16T21:57:29.294 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.289+0000 7fd8777fe640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd864066410 0x7fd8640688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:29.294 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.289+0000 7fd8777fe640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd864066410 0x7fd8640688d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd868000f70 tx=0x7fd868009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:29.295 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.293+0000 7fd85ffff640 1 -- 172.21.15.28:0/1348165143 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fd86c0507c0 con 0x7fd87813f630 2023-12-16T21:57:29.447 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.445+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd87813a360 con 0x7fd87813f630 2023-12-16T21:57:29.449 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.445+0000 7fd85ffff640 1 -- 172.21.15.28:0/1348165143 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+7542 (secure 0 0 0) 0x7fd86c017090 con 0x7fd87813f630 2023-12-16T21:57:29.449 INFO:teuthology.misc.health.smithi028.stdout: 2023-12-16T21:57:29.449 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":14,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:28.427758+0000","last_up_change":"2023-12-16T21:57:28.427758+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:29.451 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd864066410 msgr2=0x7fd8640688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:29.451 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd864066410 0x7fd8640688d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd868000f70 tx=0x7fd868009040 comp rx=0 tx=0).stop 2023-12-16T21:57:29.451 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813f630 msgr2=0x7fd878154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:29.451 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813f630 0x7fd878154300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00c3c0 tx=0x7fd86c00c880 comp rx=0 tx=0).stop 2023-12-16T21:57:29.451 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 shutdown_connections 2023-12-16T21:57:29.451 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd864066410 0x7fd8640688d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:29.451 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8780b1540 0x7fd878136410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:29.452 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd87813f630 0x7fd878154300 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:29.452 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 --2- 172.21.15.28:0/1348165143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd87813cfa0 0x7fd8780afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:29.452 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 >> 172.21.15.28:0/1348165143 conn(0x7fd87806c5e0 msgr2=0x7fd8780664c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:29.452 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 shutdown_connections 2023-12-16T21:57:29.452 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:29.449+0000 7fd87df02640 1 -- 172.21.15.28:0/1348165143 wait complete. 2023-12-16T21:57:29.467 DEBUG:teuthology.misc:1 of 8 OSDs are up 2023-12-16T21:57:29.670 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-16T21:57:29.668+0000 7f0e00b3e640 -1 osd.4 0 waiting for initial osdmap 2023-12-16T21:57:29.679 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-16T21:57:29.680+0000 7f66964b3640 -1 osd.5 0 waiting for initial osdmap 2023-12-16T21:57:29.697 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-16T21:57:29.696+0000 7f0df918b640 -1 osd.4 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:57:29.697 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-16T21:57:29.696+0000 7f5af6bb8640 -1 osd.6 0 waiting for initial osdmap 2023-12-16T21:57:29.707 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-16T21:57:29.708+0000 7f668eb00640 -1 osd.5 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:57:29.715 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-16T21:57:29.716+0000 7f5aef205640 -1 osd.6 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:57:29.781 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-16T21:57:29.780+0000 7f702d757640 -1 osd.7 0 waiting for initial osdmap 2023-12-16T21:57:29.811 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-16T21:57:29.812+0000 7f702456e640 -1 osd.7 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:57:36.274 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:36.403 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a9960f640 1 Processor -- start 2023-12-16T21:57:36.403 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a9960f640 1 -- start start 2023-12-16T21:57:36.404 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a9960f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a940b41d0 0x7f6a94071d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.404 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a9960f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a940705e0 con 0x7f6a9415b4b0 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a9960f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a940708c0 con 0x7f6a940b45b0 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a9960f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a94071680 con 0x7f6a940b41d0 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a92ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a940b41d0 0x7f6a94071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a92ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a940b41d0 0x7f6a94071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40380/0 (socket says 172.21.15.28:40380) 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a937fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6a9415b4b0 0x7f6a9415b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42130/0 (socket says 172.21.15.28:42130) 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a92ffd640 1 -- 172.21.15.28:0/1389752807 learned_addr learned my addr 172.21.15.28:0/1389752807 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a92ffd640 1 -- 172.21.15.28:0/1389752807 >> v1:172.21.15.122:6789/0 conn(0x7f6a940b45b0 legacy=0x7f6a9415f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a92ffd640 1 -- 172.21.15.28:0/1389752807 >> v1:172.21.15.28:6789/0 conn(0x7f6a9415b4b0 legacy=0x7f6a9415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a91ffb640 1 -- 172.21.15.28:0/1389752807 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994243296 0 0) 0x7f6a940705e0 con 0x7f6a9415b4b0 2023-12-16T21:57:36.405 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.401+0000 7f6a92ffd640 1 -- 172.21.15.28:0/1389752807 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a94070ba0 con 0x7f6a940b41d0 2023-12-16T21:57:36.406 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a92ffd640 1 --2- 172.21.15.28:0/1389752807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b41d0 0x7f6a94071d00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009870 tx=0x7f6a8801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72860cafb369455 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:36.408 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a91ffb640 1 -- 172.21.15.28:0/1389752807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a880aa670 con 0x7f6a940b41d0 2023-12-16T21:57:36.408 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a91ffb640 1 -- 172.21.15.28:0/1389752807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9415c840 con 0x7f6a940b41d0 2023-12-16T21:57:36.408 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a91ffb640 1 -- 172.21.15.28:0/1389752807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a880aa9f0 con 0x7f6a940b41d0 2023-12-16T21:57:36.408 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- 172.21.15.28:0/1389752807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b41d0 msgr2=0x7f6a94071d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.408 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 --2- 172.21.15.28:0/1389752807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b41d0 0x7f6a94071d00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a88009870 tx=0x7f6a8801c510 comp rx=0 tx=0).stop 2023-12-16T21:57:36.408 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- 172.21.15.28:0/1389752807 shutdown_connections 2023-12-16T21:57:36.408 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 --2- 172.21.15.28:0/1389752807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b41d0 0x7f6a94071d00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.408 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- 172.21.15.28:0/1389752807 >> 172.21.15.28:0/1389752807 conn(0x7f6a9406c5e0 msgr2=0x7f6a9406c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:36.409 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- 172.21.15.28:0/1389752807 shutdown_connections 2023-12-16T21:57:36.409 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- 172.21.15.28:0/1389752807 wait complete. 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 Processor -- start 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- start start 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a940b41d0 0x7f6a94140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b45b0 0x7f6a94150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9415b4b0 0x7f6a9413d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9414d6c0 con 0x7f6a940b45b0 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a941512a0 con 0x7f6a9415b4b0 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a9960f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9414ad00 con 0x7f6a940b41d0 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b45b0 0x7f6a94150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.411 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b45b0 0x7f6a94150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40390/0 (socket says 172.21.15.28:40390) 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a927fc640 1 -- 172.21.15.28:0/1560490948 learned_addr learned my addr 172.21.15.28:0/1560490948 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a92ffd640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a940b41d0 0x7f6a94140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a937fe640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9415b4b0 0x7f6a9413d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a927fc640 1 -- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a940b41d0 msgr2=0x7f6a94140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a927fc640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a940b41d0 0x7f6a94140020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a927fc640 1 -- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9415b4b0 msgr2=0x7f6a9413d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a927fc640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9415b4b0 0x7f6a9413d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a927fc640 1 -- 172.21.15.28:0/1560490948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a880094d0 con 0x7f6a940b45b0 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a937fe640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9415b4b0 0x7f6a9413d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.405+0000 7f6a92ffd640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a940b41d0 0x7f6a94140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a927fc640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b45b0 0x7f6a94150d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a8400c500 tx=0x7f6a8400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a7b7fe640 1 -- 172.21.15.28:0/1560490948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84005220 con 0x7f6a940b45b0 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a7b7fe640 1 -- 172.21.15.28:0/1560490948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a84005380 con 0x7f6a940b45b0 2023-12-16T21:57:36.412 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9414d8a0 con 0x7f6a940b45b0 2023-12-16T21:57:36.413 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a7b7fe640 1 -- 172.21.15.28:0/1560490948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84007270 con 0x7f6a940b45b0 2023-12-16T21:57:36.413 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9414afa0 con 0x7f6a940b45b0 2023-12-16T21:57:36.414 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a940694b0 con 0x7f6a940b45b0 2023-12-16T21:57:36.414 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a7b7fe640 1 -- 172.21.15.28:0/1560490948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6a840073d0 con 0x7f6a940b45b0 2023-12-16T21:57:36.414 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a7b7fe640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.414 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a7b7fe640 1 -- 172.21.15.28:0/1560490948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6a84083410 con 0x7f6a940b45b0 2023-12-16T21:57:36.414 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a92ffd640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.414 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.409+0000 7f6a92ffd640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a880ae000 tx=0x7f6a88008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:36.417 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.413+0000 7f6a7b7fe640 1 -- 172.21.15.28:0/1560490948 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6a84050bf0 con 0x7f6a940b45b0 2023-12-16T21:57:36.572 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.569+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a941576f0 con 0x7f6a940b45b0 2023-12-16T21:57:36.574 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.569+0000 7f6a7b7fe640 1 -- 172.21.15.28:0/1560490948 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f6a84054210 con 0x7f6a940b45b0 2023-12-16T21:57:36.574 INFO:teuthology.misc.health.smithi028.stdout: 2023-12-16T21:57:36.574 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":19,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:33.469854+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:36.576 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 msgr2=0x7f6a80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.576 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a880ae000 tx=0x7f6a88008040 comp rx=0 tx=0).stop 2023-12-16T21:57:36.576 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b45b0 msgr2=0x7f6a94150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.576 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b45b0 0x7f6a94150d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a8400c500 tx=0x7f6a8400c9c0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.577 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 shutdown_connections 2023-12-16T21:57:36.577 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.577 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9415b4b0 0x7f6a9413d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.577 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a940b45b0 0x7f6a94150d60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.577 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 --2- 172.21.15.28:0/1560490948 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a940b41d0 0x7f6a94140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.577 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 >> 172.21.15.28:0/1560490948 conn(0x7f6a9406c5e0 msgr2=0x7f6a9415cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:36.577 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 shutdown_connections 2023-12-16T21:57:36.577 INFO:teuthology.misc.health.smithi028.stderr:2023-12-16T21:57:36.573+0000 7f6a9960f640 1 -- 172.21.15.28:0/1560490948 wait complete. 2023-12-16T21:57:36.593 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-16T21:57:36.593 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-16T21:57:36.593 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-16T21:57:36.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-16T21:57:36.596 INFO:tasks.ceph.mds.a:Started 2023-12-16T21:57:36.596 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-16T21:57:36.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-16T21:57:36.598 INFO:tasks.ceph.mds.c:Started 2023-12-16T21:57:36.598 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-16T21:57:36.598 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-16T21:57:36.601 INFO:tasks.ceph.mds.b:Started 2023-12-16T21:57:36.601 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-16T21:57:36.601 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-16T21:57:36.604 INFO:tasks.ceph.mds.d:Started 2023-12-16T21:57:36.604 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-12-16T21:57:36.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:36.671 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2023-12-16T21:57:36.671 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2023-12-16T21:57:36.682 INFO:tasks.ceph.mds.d.smithi122.stdout:starting mds.d at 2023-12-16T21:57:36.683 INFO:tasks.ceph.mds.b.smithi122.stdout:starting mds.b at 2023-12-16T21:57:36.722 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f335395b640 1 Processor -- start 2023-12-16T21:57:36.722 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f335395b640 1 -- start start 2023-12-16T21:57:36.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f335395b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f334c06fc20 0x7f334c0cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f335395b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c06da10 0x7f334c0ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f335395b640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c06ee30 con 0x7f334c15a1b0 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f335395b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334c0cdb60 con 0x7f334c06fc20 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f335395b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c06f080 con 0x7f334c06da10 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f33516f7640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f334c15a1b0 0x7f334c06d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42150/0 (socket says 172.21.15.28:42150) 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f33516f7640 1 -- 172.21.15.28:0/1574889721 learned_addr learned my addr 172.21.15.28:0/1574889721 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f3351ef8640 1 --2- 172.21.15.28:0/1574889721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c06da10 0x7f334c0ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f3350ef6640 1 --2- 172.21.15.28:0/1574889721 >> v2:172.21.15.122:3300/0 conn(0x7f334c06fc20 0x7f334c0cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f3343fff640 1 -- 172.21.15.28:0/1574889721 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375501799 0 0) 0x7f334c06ee30 con 0x7f334c15a1b0 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f3343fff640 1 -- 172.21.15.28:0/1574889721 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3338003530 con 0x7f334c15a1b0 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f3351ef8640 1 -- 172.21.15.28:0/1574889721 >> v2:172.21.15.122:3300/0 conn(0x7f334c06fc20 msgr2=0x7f334c0cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f3351ef8640 1 --2- 172.21.15.28:0/1574889721 >> v2:172.21.15.122:3300/0 conn(0x7f334c06fc20 0x7f334c0cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f3351ef8640 1 -- 172.21.15.28:0/1574889721 >> v1:172.21.15.28:6789/0 conn(0x7f334c15a1b0 legacy=0x7f334c06d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.721+0000 7f3351ef8640 1 -- 172.21.15.28:0/1574889721 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0cd7f0 con 0x7f334c06da10 2023-12-16T21:57:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3351ef8640 1 --2- 172.21.15.28:0/1574889721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c06da10 0x7f334c0ce530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3344009d90 tx=0x7f334401c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9073467ebe9cd2b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3343fff640 1 -- 172.21.15.28:0/1574889721 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33440a5a30 con 0x7f334c06da10 2023-12-16T21:57:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3343fff640 1 -- 172.21.15.28:0/1574889721 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33440a5b90 con 0x7f334c06da10 2023-12-16T21:57:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3343fff640 1 -- 172.21.15.28:0/1574889721 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33440ad440 con 0x7f334c06da10 2023-12-16T21:57:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- 172.21.15.28:0/1574889721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c06da10 msgr2=0x7f334c0ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 --2- 172.21.15.28:0/1574889721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c06da10 0x7f334c0ce530 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3344009d90 tx=0x7f334401c950 comp rx=0 tx=0).stop 2023-12-16T21:57:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- 172.21.15.28:0/1574889721 shutdown_connections 2023-12-16T21:57:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 --2- 172.21.15.28:0/1574889721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c06da10 0x7f334c0ce530 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 --2- 172.21.15.28:0/1574889721 >> v2:172.21.15.122:3300/0 conn(0x7f334c06fc20 0x7f334c0cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- 172.21.15.28:0/1574889721 >> 172.21.15.28:0/1574889721 conn(0x7f334c0cfe90 msgr2=0x7f334c0d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- 172.21.15.28:0/1574889721 shutdown_connections 2023-12-16T21:57:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- 172.21.15.28:0/1574889721 wait complete. 2023-12-16T21:57:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 Processor -- start 2023-12-16T21:57:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- start start 2023-12-16T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f334c06da10 0x7f334c0b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c15a1b0 0x7f334c15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c147710 0x7f334c151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c15b930 con 0x7f334c15a1b0 2023-12-16T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c15c750 con 0x7f334c06da10 2023-12-16T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f335395b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c147f40 con 0x7f334c147710 2023-12-16T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c15a1b0 0x7f334c15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c15a1b0 0x7f334c15b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40410/0 (socket says 172.21.15.28:40410) 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 -- 172.21.15.28:0/3846218432 learned_addr learned my addr 172.21.15.28:0/3846218432 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3351ef8640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c147710 0x7f334c151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f33516f7640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f334c06da10 0x7f334c0b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 -- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c147710 msgr2=0x7f334c151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c147710 0x7f334c151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 -- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f334c06da10 msgr2=0x7f334c0b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f334c06da10 0x7f334c0b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 -- 172.21.15.28:0/3846218432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344009a70 con 0x7f334c15a1b0 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f33516f7640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f334c06da10 0x7f334c0b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3350ef6640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c15a1b0 0x7f334c15b3f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f333c00e930 tx=0x7f333c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3341ffb640 1 -- 172.21.15.28:0/3846218432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c009690 con 0x7f334c15a1b0 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3341ffb640 1 -- 172.21.15.28:0/3846218432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c00ca00 con 0x7f334c15a1b0 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.725+0000 7f3341ffb640 1 -- 172.21.15.28:0/3846218432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c00cd70 con 0x7f334c15a1b0 2023-12-16T21:57:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.729+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c15bb40 con 0x7f334c15a1b0 2023-12-16T21:57:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.729+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c15c100 con 0x7f334c15a1b0 2023-12-16T21:57:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.729+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334c0af920 con 0x7f334c15a1b0 2023-12-16T21:57:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.729+0000 7f3341ffb640 1 -- 172.21.15.28:0/3846218432 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f333c013470 con 0x7f334c15a1b0 2023-12-16T21:57:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.729+0000 7f3341ffb640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f33300664d0 0x7f3330068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.729+0000 7f3341ffb640 1 -- 172.21.15.28:0/3846218432 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f333c083d80 con 0x7f334c15a1b0 2023-12-16T21:57:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.729+0000 7f33516f7640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f33300664d0 0x7f3330068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.729+0000 7f33516f7640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f33300664d0 0x7f3330068990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f33480011f0 tx=0x7f3348009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.733+0000 7f3341ffb640 1 -- 172.21.15.28:0/3846218432 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f333c051610 con 0x7f334c15a1b0 2023-12-16T21:57:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.889+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f334c15c8c0 con 0x7f334c15a1b0 2023-12-16T21:57:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.889+0000 7f3341ffb640 1 -- 172.21.15.28:0/3846218432 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f333c017090 con 0x7f334c15a1b0 2023-12-16T21:57:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:36.893 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:33.469854+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f33300664d0 msgr2=0x7f3330068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f33300664d0 0x7f3330068990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f33480011f0 tx=0x7f3348009040 comp rx=0 tx=0).stop 2023-12-16T21:57:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c15a1b0 msgr2=0x7f334c15b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c15a1b0 0x7f334c15b3f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f333c00e930 tx=0x7f333c00edf0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 shutdown_connections 2023-12-16T21:57:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f33300664d0 0x7f3330068990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f334c147710 0x7f334c151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f334c15a1b0 0x7f334c15b3f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 --2- 172.21.15.28:0/3846218432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f334c06da10 0x7f334c0b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 >> 172.21.15.28:0/3846218432 conn(0x7f334c0cfe90 msgr2=0x7f334c06d300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 shutdown_connections 2023-12-16T21:57:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:36.893+0000 7f335395b640 1 -- 172.21.15.28:0/3846218432 wait complete. 2023-12-16T21:57:36.912 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:57:36.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:57:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd2cea5640 1 Processor -- start 2023-12-16T21:57:37.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd2cea5640 1 -- start start 2023-12-16T21:57:37.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd2cea5640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd28158ca0 0x7fdd2815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd2cea5640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd280b42d0 0x7fdd280b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd2cea5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd280b4c40 con 0x7fdd280cba10 2023-12-16T21:57:37.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd2cea5640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2815b670 con 0x7fdd280b42d0 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd2cea5640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2815c510 con 0x7fdd28158ca0 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd26d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd280b42d0 0x7fdd280b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd26d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd280b42d0 0x7fdd280b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40426/0 (socket says 172.21.15.28:40426) 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd26d9d640 1 -- 172.21.15.28:0/2773013978 learned_addr learned my addr 172.21.15.28:0/2773013978 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd2659c640 1 --2- 172.21.15.28:0/2773013978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd28158ca0 0x7fdd2815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd26d9d640 1 -- 172.21.15.28:0/2773013978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd28158ca0 msgr2=0x7fdd2815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd26d9d640 1 --2- 172.21.15.28:0/2773013978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd28158ca0 0x7fdd2815eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd26d9d640 1 -- 172.21.15.28:0/2773013978 >> v1:172.21.15.122:6789/0 conn(0x7fdd280cba10 legacy=0x7fdd2815be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.033+0000 7fdd26d9d640 1 -- 172.21.15.28:0/2773013978 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2815f410 con 0x7fdd280b42d0 2023-12-16T21:57:37.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd26d9d640 1 --2- 172.21.15.28:0/2773013978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd280b4700 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009e50 tx=0x7fdd2001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27acc4b0e0cc6a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2559a640 1 -- 172.21.15.28:0/2773013978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd200a58a0 con 0x7fdd280b42d0 2023-12-16T21:57:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2559a640 1 -- 172.21.15.28:0/2773013978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd200a5a00 con 0x7fdd280b42d0 2023-12-16T21:57:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2559a640 1 -- 172.21.15.28:0/2773013978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd200a5d90 con 0x7fdd280b42d0 2023-12-16T21:57:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- 172.21.15.28:0/2773013978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 msgr2=0x7fdd280b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/2773013978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd280b4700 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009e50 tx=0x7fdd2001c9e0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- 172.21.15.28:0/2773013978 shutdown_connections 2023-12-16T21:57:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/2773013978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd280b4700 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/2773013978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd28158ca0 0x7fdd2815eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- 172.21.15.28:0/2773013978 >> 172.21.15.28:0/2773013978 conn(0x7fdd280b0cc0 msgr2=0x7fdd280b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- 172.21.15.28:0/2773013978 shutdown_connections 2023-12-16T21:57:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- 172.21.15.28:0/2773013978 wait complete. 2023-12-16T21:57:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 Processor -- start 2023-12-16T21:57:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- start start 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd281555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd280cba10 0x7fdd28161940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd28158ca0 0x7fdd28161e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28155ae0 con 0x7fdd280b42d0 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28153360 con 0x7fdd28158ca0 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2cea5640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28153640 con 0x7fdd280cba10 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd281555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd281555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40436/0 (socket says 172.21.15.28:40436) 2023-12-16T21:57:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.037+0000 7fdd2659c640 1 -- 172.21.15.28:0/1593363202 learned_addr learned my addr 172.21.15.28:0/1593363202 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd25d9b640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd280cba10 0x7fdd28161940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd26d9d640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd28158ca0 0x7fdd28161e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2659c640 1 -- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd280cba10 msgr2=0x7fdd28161940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2659c640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd280cba10 0x7fdd28161940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2659c640 1 -- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd28158ca0 msgr2=0x7fdd28161e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2659c640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd28158ca0 0x7fdd28161e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2659c640 1 -- 172.21.15.28:0/1593363202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20009b30 con 0x7fdd280b42d0 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd25d9b640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd280cba10 0x7fdd28161940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd26d9d640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd28158ca0 0x7fdd28161e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2659c640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd281555a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00d800 tx=0x7fdd1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd16ffd640 1 -- 172.21.15.28:0/1593363202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00b6c0 con 0x7fdd280b42d0 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd16ffd640 1 -- 172.21.15.28:0/1593363202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c00b820 con 0x7fdd280b42d0 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd16ffd640 1 -- 172.21.15.28:0/1593363202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00bb90 con 0x7fdd280b42d0 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd28153920 con 0x7fdd280b42d0 2023-12-16T21:57:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd28153ee0 con 0x7fdd280b42d0 2023-12-16T21:57:37.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd281540a0 con 0x7fdd280b42d0 2023-12-16T21:57:37.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd16ffd640 1 -- 172.21.15.28:0/1593363202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fdd1c00bcf0 con 0x7fdd280b42d0 2023-12-16T21:57:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd16ffd640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdd08066470 0x7fdd08068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd16ffd640 1 -- 172.21.15.28:0/1593363202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdd1c084060 con 0x7fdd280b42d0 2023-12-16T21:57:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd25d9b640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdd08066470 0x7fdd08068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.041+0000 7fdd25d9b640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdd08066470 0x7fdd08068930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdd280b51d0 tx=0x7fdd18009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:37.049 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.045+0000 7fdd16ffd640 1 -- 172.21.15.28:0/1593363202 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fdd1c051830 con 0x7fdd280b42d0 2023-12-16T21:57:37.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.201+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd280693d0 con 0x7fdd280b42d0 2023-12-16T21:57:37.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd16ffd640 1 -- 172.21.15.28:0/1593363202 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd1c054e50 con 0x7fdd280b42d0 2023-12-16T21:57:37.206 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:57:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdd08066470 msgr2=0x7fdd08068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdd08066470 0x7fdd08068930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdd280b51d0 tx=0x7fdd18009040 comp rx=0 tx=0).stop 2023-12-16T21:57:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 msgr2=0x7fdd281555a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd281555a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00d800 tx=0x7fdd1c00dcc0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 shutdown_connections 2023-12-16T21:57:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdd08066470 0x7fdd08068930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd28158ca0 0x7fdd28161e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdd280cba10 0x7fdd28161940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 --2- 172.21.15.28:0/1593363202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd280b42d0 0x7fdd281555a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 >> 172.21.15.28:0/1593363202 conn(0x7fdd280b0cc0 msgr2=0x7fdd28159c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 shutdown_connections 2023-12-16T21:57:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.205+0000 7fdd2cea5640 1 -- 172.21.15.28:0/1593363202 wait complete. 2023-12-16T21:57:37.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:57:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83cdf5a640 1 Processor -- start 2023-12-16T21:57:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83cdf5a640 1 -- start start 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83cdf5a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83c806cdb0 0x7f83c8150250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83cdf5a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806d190 0x7f83c8150790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83cdf5a640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c8070460 con 0x7f83c8159da0 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83cdf5a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c8150cd0 con 0x7f83c806cdb0 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83cdf5a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c815a900 con 0x7f83c806d190 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c77fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83c806cdb0 0x7f83c8150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c7fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f83c8159da0 0x7f83c815a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42160/0 (socket says 172.21.15.28:42160) 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c7fff640 1 -- 172.21.15.28:0/3607935446 learned_addr learned my addr 172.21.15.28:0/3607935446 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c77fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83c806cdb0 0x7f83c8150250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40446/0 (socket says 172.21.15.28:40446) 2023-12-16T21:57:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c6ffd640 1 --2- 172.21.15.28:0/3607935446 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806d190 0x7f83c8150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c77fe640 1 -- 172.21.15.28:0/3607935446 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806d190 msgr2=0x7f83c8150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c77fe640 1 --2- 172.21.15.28:0/3607935446 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806d190 0x7f83c8150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c77fe640 1 -- 172.21.15.28:0/3607935446 >> v1:172.21.15.28:6789/0 conn(0x7f83c8159da0 legacy=0x7f83c815a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.337+0000 7f83c77fe640 1 -- 172.21.15.28:0/3607935446 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c815ab20 con 0x7f83c806cdb0 2023-12-16T21:57:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c77fe640 1 --2- 172.21.15.28:0/3607935446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c806cdb0 0x7f83c8150250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83b80097c0 tx=0x7f83b801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f84bef6aa191163 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c67fc640 1 -- 172.21.15.28:0/3607935446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b80ab600 con 0x7f83c806cdb0 2023-12-16T21:57:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c67fc640 1 -- 172.21.15.28:0/3607935446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b80a7050 con 0x7f83c806cdb0 2023-12-16T21:57:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c67fc640 1 -- 172.21.15.28:0/3607935446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b80ab980 con 0x7f83c806cdb0 2023-12-16T21:57:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- 172.21.15.28:0/3607935446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c806cdb0 msgr2=0x7f83c8150250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/3607935446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c806cdb0 0x7f83c8150250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83b80097c0 tx=0x7f83b801c3f0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- 172.21.15.28:0/3607935446 shutdown_connections 2023-12-16T21:57:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/3607935446 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806d190 0x7f83c8150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/3607935446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c806cdb0 0x7f83c8150250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- 172.21.15.28:0/3607935446 >> 172.21.15.28:0/3607935446 conn(0x7f83c806a530 msgr2=0x7f83c806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- 172.21.15.28:0/3607935446 shutdown_connections 2023-12-16T21:57:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- 172.21.15.28:0/3607935446 wait complete. 2023-12-16T21:57:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 Processor -- start 2023-12-16T21:57:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- start start 2023-12-16T21:57:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806cdb0 0x7f83c81469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83c806d190 0x7f83c815d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8159da0 0x7f83c815d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c8153d20 con 0x7f83c8159da0 2023-12-16T21:57:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c8134f70 con 0x7f83c806d190 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c80b2fe0 con 0x7f83c806cdb0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8159da0 0x7f83c815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8159da0 0x7f83c815d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40460/0 (socket says 172.21.15.28:40460) 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 -- 172.21.15.28:0/2338326934 learned_addr learned my addr 172.21.15.28:0/2338326934 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 -- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806cdb0 msgr2=0x7f83c81469b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c6ffd640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83c806d190 0x7f83c815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c77fe640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806cdb0 0x7f83c81469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806cdb0 0x7f83c81469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 -- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83c806d190 msgr2=0x7f83c815d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83c806d190 0x7f83c815d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 -- 172.21.15.28:0/2338326934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b8009420 con 0x7f83c8159da0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83c7fff640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8159da0 0x7f83c815d740 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f83c000ca80 tx=0x7f83c000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83affff640 1 -- 172.21.15.28:0/2338326934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c000ac30 con 0x7f83c8159da0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83affff640 1 -- 172.21.15.28:0/2338326934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c000ad90 con 0x7f83c8159da0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c8146ef0 con 0x7f83c8159da0 2023-12-16T21:57:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.341+0000 7f83affff640 1 -- 172.21.15.28:0/2338326934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0012600 con 0x7f83c8159da0 2023-12-16T21:57:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.345+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c8153760 con 0x7f83c8159da0 2023-12-16T21:57:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.345+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c8073d60 con 0x7f83c8159da0 2023-12-16T21:57:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.345+0000 7f83affff640 1 -- 172.21.15.28:0/2338326934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f83c0007070 con 0x7f83c8159da0 2023-12-16T21:57:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.345+0000 7f83affff640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83b4066470 0x7f83b4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.345+0000 7f83affff640 1 -- 172.21.15.28:0/2338326934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f83c0083980 con 0x7f83c8159da0 2023-12-16T21:57:37.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.345+0000 7f83c77fe640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83b4066470 0x7f83b4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:37.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.345+0000 7f83c77fe640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83b4066470 0x7f83b4068930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f83b80ad000 tx=0x7f83b8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:37.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.349+0000 7f83affff640 1 -- 172.21.15.28:0/2338326934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f83c0051160 con 0x7f83c8159da0 2023-12-16T21:57:37.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.529+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83c8069440 con 0x7f83c8159da0 2023-12-16T21:57:37.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.529+0000 7f83affff640 1 -- 172.21.15.28:0/2338326934 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f83c0017090 con 0x7f83c8159da0 2023-12-16T21:57:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:37.534 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-12-16T21:57:37.534 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2023-12-16T21:57:37.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83b4066470 msgr2=0x7f83b4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83b4066470 0x7f83b4068930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f83b80ad000 tx=0x7f83b8008040 comp rx=0 tx=0).stop 2023-12-16T21:57:37.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8159da0 msgr2=0x7f83c815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:37.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8159da0 0x7f83c815d740 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f83c000ca80 tx=0x7f83c000cf40 comp rx=0 tx=0).stop 2023-12-16T21:57:37.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 shutdown_connections 2023-12-16T21:57:37.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83b4066470 0x7f83b4068930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c8159da0 0x7f83c815d740 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83c806d190 0x7f83c815d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 --2- 172.21.15.28:0/2338326934 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83c806cdb0 0x7f83c81469b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 >> 172.21.15.28:0/2338326934 conn(0x7f83c806a530 msgr2=0x7f83c81503b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 shutdown_connections 2023-12-16T21:57:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:37.533+0000 7f83cdf5a640 1 -- 172.21.15.28:0/2338326934 wait complete. 2023-12-16T21:57:39.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:57:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e3db77640 1 Processor -- start 2023-12-16T21:57:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e3db77640 1 -- start start 2023-12-16T21:57:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e3db77640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e380b35b0 0x7f5e380d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e3db77640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 0x7f5e3815f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e3db77640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e380d3880 con 0x7f5e380cf780 2023-12-16T21:57:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e3db77640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e380d4090 con 0x7f5e380b35b0 2023-12-16T21:57:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e3db77640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380cdc80 con 0x7f5e380b3c70 2023-12-16T21:57:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e377fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e380b35b0 0x7f5e380d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e377fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e380b35b0 0x7f5e380d0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57294/0 (socket says 172.21.15.28:57294) 2023-12-16T21:57:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e377fe640 1 -- 172.21.15.28:0/152977500 learned_addr learned my addr 172.21.15.28:0/152977500 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e36ffd640 1 --2- 172.21.15.28:0/152977500 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 0x7f5e3815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e377fe640 1 -- 172.21.15.28:0/152977500 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 msgr2=0x7f5e3815f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e377fe640 1 --2- 172.21.15.28:0/152977500 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 0x7f5e3815f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e377fe640 1 -- 172.21.15.28:0/152977500 >> v1:172.21.15.122:6789/0 conn(0x7f5e380cf780 legacy=0x7f5e380ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.684+0000 7f5e377fe640 1 -- 172.21.15.28:0/152977500 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e380d3ad0 con 0x7f5e380b35b0 2023-12-16T21:57:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e377fe640 1 --2- 172.21.15.28:0/152977500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380d0f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009870 tx=0x7f5e2801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8003e1c97dab6df4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e367fc640 1 -- 172.21.15.28:0/152977500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e280aa620 con 0x7f5e380b35b0 2023-12-16T21:57:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e367fc640 1 -- 172.21.15.28:0/152977500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e280aa780 con 0x7f5e380b35b0 2023-12-16T21:57:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e367fc640 1 -- 172.21.15.28:0/152977500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e280aab10 con 0x7f5e380b35b0 2023-12-16T21:57:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 -- 172.21.15.28:0/152977500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 msgr2=0x7f5e380d0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 --2- 172.21.15.28:0/152977500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380d0f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009870 tx=0x7f5e2801c500 comp rx=0 tx=0).stop 2023-12-16T21:57:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 -- 172.21.15.28:0/152977500 shutdown_connections 2023-12-16T21:57:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 --2- 172.21.15.28:0/152977500 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 0x7f5e3815f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 --2- 172.21.15.28:0/152977500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380d0f50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 -- 172.21.15.28:0/152977500 >> 172.21.15.28:0/152977500 conn(0x7f5e3809c4b0 msgr2=0x7f5e380d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 -- 172.21.15.28:0/152977500 shutdown_connections 2023-12-16T21:57:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 -- 172.21.15.28:0/152977500 wait complete. 2023-12-16T21:57:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.688+0000 7f5e3db77640 1 Processor -- start 2023-12-16T21:57:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 -- start start 2023-12-16T21:57:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 0x7f5e38155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e380cf780 0x7f5e38156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3815dfe0 con 0x7f5e380b35b0 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3815d3d0 con 0x7f5e380cf780 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3815d6b0 con 0x7f5e380b3c70 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57298/0 (socket says 172.21.15.28:57298) 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 -- 172.21.15.28:0/3479003993 learned_addr learned my addr 172.21.15.28:0/3479003993 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e37fff640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e380cf780 0x7f5e38156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e36ffd640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 0x7f5e38155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 -- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 msgr2=0x7f5e38155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 0x7f5e38155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 -- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e380cf780 msgr2=0x7f5e38156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e380cf780 0x7f5e38156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 -- 172.21.15.28:0/3479003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e280094d0 con 0x7f5e380b35b0 2023-12-16T21:57:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e377fe640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380c9070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5e280ac040 tx=0x7f5e2801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e1ffff640 1 -- 172.21.15.28:0/3479003993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e280bd3f0 con 0x7f5e380b35b0 2023-12-16T21:57:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e1ffff640 1 -- 172.21.15.28:0/3479003993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e280bd570 con 0x7f5e380b35b0 2023-12-16T21:57:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3815d990 con 0x7f5e380b35b0 2023-12-16T21:57:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e1ffff640 1 -- 172.21.15.28:0/3479003993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e280bd920 con 0x7f5e380b35b0 2023-12-16T21:57:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e38157160 con 0x7f5e380b35b0 2023-12-16T21:57:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3815d7f0 con 0x7f5e380b35b0 2023-12-16T21:57:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e1ffff640 1 -- 172.21.15.28:0/3479003993 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f5e280b33f0 con 0x7f5e380b35b0 2023-12-16T21:57:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e1ffff640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5e24066470 0x7f5e24068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.692+0000 7f5e1ffff640 1 -- 172.21.15.28:0/3479003993 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5e2811e140 con 0x7f5e380b35b0 2023-12-16T21:57:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.696+0000 7f5e36ffd640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5e24066470 0x7f5e24068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.696+0000 7f5e36ffd640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5e24066470 0x7f5e24068930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5e38158eb0 tx=0x7f5e2c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.700+0000 7f5e1ffff640 1 -- 172.21.15.28:0/3479003993 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f5e3815d7f0 con 0x7f5e380b35b0 2023-12-16T21:57:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e3815d7f0 con 0x7f5e380b35b0 2023-12-16T21:57:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e1ffff640 1 -- 172.21.15.28:0/3479003993 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7f5e3815d7f0 con 0x7f5e380b35b0 2023-12-16T21:57:39.891 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-12-16T21:57:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2023-12-16T21:57:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5e24066470 msgr2=0x7f5e24068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5e24066470 0x7f5e24068930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5e38158eb0 tx=0x7f5e2c008040 comp rx=0 tx=0).stop 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 msgr2=0x7f5e380c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380c9070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5e280ac040 tx=0x7f5e2801ad40 comp rx=0 tx=0).stop 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 shutdown_connections 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5e24066470 0x7f5e24068930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5e380cf780 0x7f5e38156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5e380b3c70 0x7f5e38155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 --2- 172.21.15.28:0/3479003993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e380b35b0 0x7f5e380c9070 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 >> 172.21.15.28:0/3479003993 conn(0x7f5e3809c4b0 msgr2=0x7f5e380b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 shutdown_connections 2023-12-16T21:57:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:39.888+0000 7f5e3db77640 1 -- 172.21.15.28:0/3479003993 wait complete. 2023-12-16T21:57:39.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f776e8d3640 1 Processor -- start 2023-12-16T21:57:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f776e8d3640 1 -- start start 2023-12-16T21:57:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f776e8d3640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f776806c2c0 0x7f776806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f776e8d3640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768072480 con 0x7f7768071b90 2023-12-16T21:57:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f776e8d3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7768071470 con 0x7f7768072010 2023-12-16T21:57:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f776e8d3640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77680726e0 con 0x7f776806c2c0 2023-12-16T21:57:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f7767fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f776806c2c0 0x7f776806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f775ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7768071b90 0x7f776815ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50694/0 (socket says 172.21.15.28:50694) 2023-12-16T21:57:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f775ffff640 1 -- 172.21.15.28:0/2063930239 learned_addr learned my addr 172.21.15.28:0/2063930239 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f7767fff640 1 -- 172.21.15.28:0/2063930239 >> v1:172.21.15.122:6789/0 conn(0x7f7768072010 legacy=0x7f776815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f7767fff640 1 -- 172.21.15.28:0/2063930239 >> v1:172.21.15.28:6789/0 conn(0x7f7768071b90 legacy=0x7f776815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.032+0000 7f7767fff640 1 -- 172.21.15.28:0/2063930239 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776815fa70 con 0x7f776806c2c0 2023-12-16T21:57:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 --2- 172.21.15.28:0/2063930239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f776806c6a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f77580097c0 tx=0x7f775801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c318a75402ce782 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f77677fe640 1 -- 172.21.15.28:0/2063930239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77580ac410 con 0x7f776806c2c0 2023-12-16T21:57:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f77677fe640 1 -- 172.21.15.28:0/2063930239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77580ac570 con 0x7f776806c2c0 2023-12-16T21:57:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f77677fe640 1 -- 172.21.15.28:0/2063930239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77580ac900 con 0x7f776806c2c0 2023-12-16T21:57:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- 172.21.15.28:0/2063930239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 msgr2=0x7f776806c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 --2- 172.21.15.28:0/2063930239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f776806c6a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f77580097c0 tx=0x7f775801c3b0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- 172.21.15.28:0/2063930239 shutdown_connections 2023-12-16T21:57:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 --2- 172.21.15.28:0/2063930239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f776806c6a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- 172.21.15.28:0/2063930239 >> 172.21.15.28:0/2063930239 conn(0x7f776806a530 msgr2=0x7f776806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- 172.21.15.28:0/2063930239 shutdown_connections 2023-12-16T21:57:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- 172.21.15.28:0/2063930239 wait complete. 2023-12-16T21:57:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 Processor -- start 2023-12-16T21:57:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- start start 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f7768146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7768071b90 0x7f77681514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7768072010 0x7f7768151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7768158330 con 0x7f776806c2c0 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7768153810 con 0x7f7768072010 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776e8d3640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77681576b0 con 0x7f7768071b90 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f7768146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f7768146a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57328/0 (socket says 172.21.15.28:57328) 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 -- 172.21.15.28:0/1394130114 learned_addr learned my addr 172.21.15.28:0/1394130114 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f775ffff640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7768071b90 0x7f77681514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f776ce70640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7768072010 0x7f7768151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 -- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7768071b90 msgr2=0x7f77681514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7768071b90 0x7f77681514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 -- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7768072010 msgr2=0x7f7768151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7768072010 0x7f7768151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 -- 172.21.15.28:0/1394130114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7758009420 con 0x7f776806c2c0 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.036+0000 7f7767fff640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f7768146a70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f77580af040 tx=0x7f7758019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f77657fa640 1 -- 172.21.15.28:0/1394130114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7758006e20 con 0x7f776806c2c0 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f77657fa640 1 -- 172.21.15.28:0/1394130114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77580b5620 con 0x7f776806c2c0 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7768157990 con 0x7f776806c2c0 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f77657fa640 1 -- 172.21.15.28:0/1394130114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77580b59f0 con 0x7f776806c2c0 2023-12-16T21:57:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7768157f50 con 0x7f776806c2c0 2023-12-16T21:57:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7768069440 con 0x7f776806c2c0 2023-12-16T21:57:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f77657fa640 1 -- 172.21.15.28:0/1394130114 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f77580b3040 con 0x7f776806c2c0 2023-12-16T21:57:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f77657fa640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7748066470 0x7f7748068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f77657fa640 1 -- 172.21.15.28:0/1394130114 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f775811dd10 con 0x7f776806c2c0 2023-12-16T21:57:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f775ffff640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7748066470 0x7f7748068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.040+0000 7f775ffff640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7748066470 0x7f7748068930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f776814cda0 tx=0x7f7750009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.044+0000 7f77657fa640 1 -- 172.21.15.28:0/1394130114 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f77580eb690 con 0x7f776806c2c0 2023-12-16T21:57:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.204+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f776806d1c0 con 0x7f776806c2c0 2023-12-16T21:57:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.204+0000 7f77657fa640 1 -- 172.21.15.28:0/1394130114 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f77580b1090 con 0x7f776806c2c0 2023-12-16T21:57:40.210 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:40.210 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:33.469854+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7748066470 msgr2=0x7f7748068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7748066470 0x7f7748068930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f776814cda0 tx=0x7f7750009040 comp rx=0 tx=0).stop 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 msgr2=0x7f7768146a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f7768146a70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f77580af040 tx=0x7f7758019150 comp rx=0 tx=0).stop 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 shutdown_connections 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7748066470 0x7f7748068930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7768072010 0x7f7768151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7768071b90 0x7f77681514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 --2- 172.21.15.28:0/1394130114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f776806c2c0 0x7f7768146a70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 >> 172.21.15.28:0/1394130114 conn(0x7f776806a530 msgr2=0x7f77680aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 shutdown_connections 2023-12-16T21:57:40.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.208+0000 7f776e8d3640 1 -- 172.21.15.28:0/1394130114 wait complete. 2023-12-16T21:57:40.228 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:57:40.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:57:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e46a8640 1 Processor -- start 2023-12-16T21:57:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e46a8640 1 -- start start 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e46a8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f38dc071d90 0x7f38dc072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e46a8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc15adb0 0x7f38dc13d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e46a8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38dc13f5f0 0x7f38dc0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e46a8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38dc140520 con 0x7f38dc13f5f0 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e46a8640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38dc0cd7f0 con 0x7f38dc071d90 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e46a8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc13d2b0 con 0x7f38dc15adb0 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2c45640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38dc13f5f0 0x7f38dc0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2c45640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38dc13f5f0 0x7f38dc0ce200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57336/0 (socket says 172.21.15.28:57336) 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2c45640 1 -- 172.21.15.28:0/4157632822 learned_addr learned my addr 172.21.15.28:0/4157632822 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e1c43640 1 --2- 172.21.15.28:0/4157632822 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc15adb0 0x7f38dc13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2444640 1 --2- 172.21.15.28:0/4157632822 >> v2:172.21.15.122:3300/0 conn(0x7f38dc071d90 0x7f38dc072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2c45640 1 -- 172.21.15.28:0/4157632822 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc15adb0 msgr2=0x7f38dc13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2c45640 1 --2- 172.21.15.28:0/4157632822 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc15adb0 0x7f38dc13d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2c45640 1 -- 172.21.15.28:0/4157632822 >> v2:172.21.15.122:3300/0 conn(0x7f38dc071d90 msgr2=0x7f38dc072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2c45640 1 --2- 172.21.15.28:0/4157632822 >> v2:172.21.15.122:3300/0 conn(0x7f38dc071d90 0x7f38dc072170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.352+0000 7f38e2c45640 1 -- 172.21.15.28:0/4157632822 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc0cdd70 con 0x7f38dc13f5f0 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e2c45640 1 --2- 172.21.15.28:0/4157632822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0ce200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38dc0b1d80 tx=0x7f38d401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2d3cb72928e8f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e1442640 1 -- 172.21.15.28:0/4157632822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d40aa620 con 0x7f38dc13f5f0 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e1442640 1 -- 172.21.15.28:0/4157632822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d40aa780 con 0x7f38dc13f5f0 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e1442640 1 -- 172.21.15.28:0/4157632822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d40aab10 con 0x7f38dc13f5f0 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- 172.21.15.28:0/4157632822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 msgr2=0x7f38dc0ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 --2- 172.21.15.28:0/4157632822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0ce200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38dc0b1d80 tx=0x7f38d401c500 comp rx=0 tx=0).stop 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- 172.21.15.28:0/4157632822 shutdown_connections 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 --2- 172.21.15.28:0/4157632822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0ce200 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 --2- 172.21.15.28:0/4157632822 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc15adb0 0x7f38dc13d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 --2- 172.21.15.28:0/4157632822 >> v2:172.21.15.122:3300/0 conn(0x7f38dc071d90 0x7f38dc072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- 172.21.15.28:0/4157632822 >> 172.21.15.28:0/4157632822 conn(0x7f38dc0b0cc0 msgr2=0x7f38dc0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- 172.21.15.28:0/4157632822 shutdown_connections 2023-12-16T21:57:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- 172.21.15.28:0/4157632822 wait complete. 2023-12-16T21:57:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 Processor -- start 2023-12-16T21:57:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- start start 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc071d90 0x7f38dc0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc15adb0 0x7f38dc138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc0b34d0 con 0x7f38dc13f5f0 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc0b37b0 con 0x7f38dc15adb0 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e46a8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38dc0b3a90 con 0x7f38dc071d90 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e1c43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e1c43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0b2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57348/0 (socket says 172.21.15.28:57348) 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e1c43640 1 -- 172.21.15.28:0/2128398230 learned_addr learned my addr 172.21.15.28:0/2128398230 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e2444640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc071d90 0x7f38dc0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.356+0000 7f38e2c45640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc15adb0 0x7f38dc138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e1c43640 1 -- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc071d90 msgr2=0x7f38dc0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e1c43640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc071d90 0x7f38dc0b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e1c43640 1 -- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc15adb0 msgr2=0x7f38dc138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e1c43640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc15adb0 0x7f38dc138100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e1c43640 1 -- 172.21.15.28:0/2128398230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d40094d0 con 0x7f38dc13f5f0 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e1c43640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0b2bc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00e870 tx=0x7f38cc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38d2ffd640 1 -- 172.21.15.28:0/2128398230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc009510 con 0x7f38dc13f5f0 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38d2ffd640 1 -- 172.21.15.28:0/2128398230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38cc00eec0 con 0x7f38dc13f5f0 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38dc0b3d70 con 0x7f38dc13f5f0 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38d2ffd640 1 -- 172.21.15.28:0/2128398230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38cc00cb40 con 0x7f38dc13f5f0 2023-12-16T21:57:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38dc0b3100 con 0x7f38dc13f5f0 2023-12-16T21:57:40.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38dc0693d0 con 0x7f38dc13f5f0 2023-12-16T21:57:40.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38d2ffd640 1 -- 172.21.15.28:0/2128398230 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f38cc00cca0 con 0x7f38dc13f5f0 2023-12-16T21:57:40.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38d2ffd640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38c0066470 0x7f38c0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38d2ffd640 1 -- 172.21.15.28:0/2128398230 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f38cc083bc0 con 0x7f38dc13f5f0 2023-12-16T21:57:40.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e2444640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38c0066470 0x7f38c0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.360+0000 7f38e2444640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38c0066470 0x7f38c0068930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f38d8003fd0 tx=0x7f38d80039c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.364+0000 7f38d2ffd640 1 -- 172.21.15.28:0/2128398230 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f38cc051390 con 0x7f38dc13f5f0 2023-12-16T21:57:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38dc06e710 con 0x7f38dc13f5f0 2023-12-16T21:57:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38d2ffd640 1 -- 172.21.15.28:0/2128398230 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38cc0549b0 con 0x7f38dc13f5f0 2023-12-16T21:57:40.528 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:57:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38c0066470 msgr2=0x7f38c0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38c0066470 0x7f38c0068930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f38d8003fd0 tx=0x7f38d80039c0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 msgr2=0x7f38dc0b2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0b2bc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38cc00e870 tx=0x7f38cc00ec70 comp rx=0 tx=0).stop 2023-12-16T21:57:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 shutdown_connections 2023-12-16T21:57:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38c0066470 0x7f38c0068930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38dc15adb0 0x7f38dc138100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38dc13f5f0 0x7f38dc0b2bc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 --2- 172.21.15.28:0/2128398230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38dc071d90 0x7f38dc0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 >> 172.21.15.28:0/2128398230 conn(0x7f38dc0b0cc0 msgr2=0x7f38dc13b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 shutdown_connections 2023-12-16T21:57:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.524+0000 7f38e46a8640 1 -- 172.21.15.28:0/2128398230 wait complete. 2023-12-16T21:57:40.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-16T21:57:40.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.664+0000 7f4a16fec640 1 Processor -- start 2023-12-16T21:57:40.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.664+0000 7f4a16fec640 1 -- start start 2023-12-16T21:57:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a16fec640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4a10144840 0x7f4a1006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a16fec640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10070b70 0x7f4a1006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a16fec640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a100710e0 con 0x7f4a10072180 2023-12-16T21:57:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a16fec640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1015f010 con 0x7f4a10144840 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a16fec640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a100713c0 con 0x7f4a10070b70 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a15589640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10070b70 0x7f4a1006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a14d88640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4a10144840 0x7f4a1006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a0ffff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4a10072180 0x7f4a10070460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:43580/0 (socket says 172.21.15.28:43580) 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a0ffff640 1 -- 172.21.15.28:0/1057787540 learned_addr learned my addr 172.21.15.28:0/1057787540 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a15589640 1 -- 172.21.15.28:0/1057787540 >> v2:172.21.15.122:3300/0 conn(0x7f4a10144840 msgr2=0x7f4a1006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a15589640 1 --2- 172.21.15.28:0/1057787540 >> v2:172.21.15.122:3300/0 conn(0x7f4a10144840 0x7f4a1006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a15589640 1 -- 172.21.15.28:0/1057787540 >> v1:172.21.15.122:6789/0 conn(0x7f4a10072180 legacy=0x7f4a10070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a15589640 1 -- 172.21.15.28:0/1057787540 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a100716a0 con 0x7f4a10070b70 2023-12-16T21:57:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a15589640 1 --2- 172.21.15.28:0/1057787540 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10070b70 0x7f4a1006c800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a08009d90 tx=0x7f4a0801c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa20750d494a3874 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a0f7fe640 1 -- 172.21.15.28:0/1057787540 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a080a59e0 con 0x7f4a10070b70 2023-12-16T21:57:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a0f7fe640 1 -- 172.21.15.28:0/1057787540 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a080a5b40 con 0x7f4a10070b70 2023-12-16T21:57:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.668+0000 7f4a0f7fe640 1 -- 172.21.15.28:0/1057787540 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a080ad440 con 0x7f4a10070b70 2023-12-16T21:57:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- 172.21.15.28:0/1057787540 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10070b70 msgr2=0x7f4a1006c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 --2- 172.21.15.28:0/1057787540 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10070b70 0x7f4a1006c800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4a08009d90 tx=0x7f4a0801c920 comp rx=0 tx=0).stop 2023-12-16T21:57:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- 172.21.15.28:0/1057787540 shutdown_connections 2023-12-16T21:57:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 --2- 172.21.15.28:0/1057787540 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10070b70 0x7f4a1006c800 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 --2- 172.21.15.28:0/1057787540 >> v2:172.21.15.122:3300/0 conn(0x7f4a10144840 0x7f4a1006c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- 172.21.15.28:0/1057787540 >> 172.21.15.28:0/1057787540 conn(0x7f4a1006a530 msgr2=0x7f4a1006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- 172.21.15.28:0/1057787540 shutdown_connections 2023-12-16T21:57:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- 172.21.15.28:0/1057787540 wait complete. 2023-12-16T21:57:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 Processor -- start 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- start start 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a10070b70 0x7f4a1015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a10072180 0x7f4a101277a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10144840 0x7f4a10164140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1013d380 con 0x7f4a10072180 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1015af00 con 0x7f4a10070b70 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1013c700 con 0x7f4a10144840 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a10072180 0x7f4a101277a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a10072180 0x7f4a101277a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57354/0 (socket says 172.21.15.28:57354) 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 -- 172.21.15.28:0/3525119679 learned_addr learned my addr 172.21.15.28:0/3525119679 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a15589640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10144840 0x7f4a10164140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a14d88640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a10070b70 0x7f4a1015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 -- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10144840 msgr2=0x7f4a10164140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10144840 0x7f4a10164140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 -- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a10070b70 msgr2=0x7f4a1015a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a10070b70 0x7f4a1015a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 -- 172.21.15.28:0/3525119679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a08009a70 con 0x7f4a10072180 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0ffff640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a10072180 0x7f4a101277a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000c460 tx=0x7f4a0000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0d7fa640 1 -- 172.21.15.28:0/3525119679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a00005cf0 con 0x7f4a10072180 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0d7fa640 1 -- 172.21.15.28:0/3525119679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a00005e50 con 0x7f4a10072180 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1013c9e0 con 0x7f4a10072180 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a0d7fa640 1 -- 172.21.15.28:0/3525119679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a00007290 con 0x7f4a10072180 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.672+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1013cfa0 con 0x7f4a10072180 2023-12-16T21:57:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.676+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a100cb1d0 con 0x7f4a10072180 2023-12-16T21:57:40.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.676+0000 7f4a0d7fa640 1 -- 172.21.15.28:0/3525119679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4a000073f0 con 0x7f4a10072180 2023-12-16T21:57:40.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.676+0000 7f4a0d7fa640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f49fc066470 0x7f49fc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.676+0000 7f4a0d7fa640 1 -- 172.21.15.28:0/3525119679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4a00083090 con 0x7f4a10072180 2023-12-16T21:57:40.682 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.676+0000 7f4a14d88640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f49fc066470 0x7f49fc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.682 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.676+0000 7f4a14d88640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f49fc066470 0x7f49fc068930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009a50 tx=0x7f4a04008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.680+0000 7f4a0d7fa640 1 -- 172.21.15.28:0/3525119679 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4a000507d0 con 0x7f4a10072180 2023-12-16T21:57:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4a10069440 con 0x7f4a10072180 2023-12-16T21:57:40.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a0d7fa640 1 -- 172.21.15.28:0/3525119679 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4a00017060 con 0x7f4a10072180 2023-12-16T21:57:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:40.840 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2023-12-16T21:57:40.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:57:40.841 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2023-12-16T21:57:40.841 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2023-12-16T21:57:40.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:57:40.841 INFO:teuthology.orchestra.run.smithi028.stdout:] 2023-12-16T21:57:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f49fc066470 msgr2=0x7f49fc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f49fc066470 0x7f49fc068930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009a50 tx=0x7f4a04008040 comp rx=0 tx=0).stop 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a10072180 msgr2=0x7f4a101277a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a10072180 0x7f4a101277a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000c460 tx=0x7f4a0000c920 comp rx=0 tx=0).stop 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 shutdown_connections 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f49fc066470 0x7f49fc068930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a10144840 0x7f4a10164140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a10072180 0x7f4a101277a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 --2- 172.21.15.28:0/3525119679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a10070b70 0x7f4a1015a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 >> 172.21.15.28:0/3525119679 conn(0x7f4a1006a530 msgr2=0x7f4a1006c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 shutdown_connections 2023-12-16T21:57:40.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.836+0000 7f4a16fec640 1 -- 172.21.15.28:0/3525119679 wait complete. 2023-12-16T21:57:40.857 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-16T21:57:40.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-16T21:57:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f3d32640 1 Processor -- start 2023-12-16T21:57:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f3d32640 1 -- start start 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f3d32640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec159ba0 0x7f64ec0c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f3d32640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64ec0cd570 0x7f64ec070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f3d32640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ec159f80 con 0x7f64ec0cb0c0 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f3d32640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64ec0cb520 con 0x7f64ec0cd570 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f3d32640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec0ce460 con 0x7f64ec159ba0 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f22cf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64ec0cd570 0x7f64ec070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f22cf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64ec0cd570 0x7f64ec070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57364/0 (socket says 172.21.15.28:57364) 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f22cf640 1 -- 172.21.15.28:0/1323295641 learned_addr learned my addr 172.21.15.28:0/1323295641 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f1ace640 1 --2- 172.21.15.28:0/1323295641 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec159ba0 0x7f64ec0c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f22cf640 1 -- 172.21.15.28:0/1323295641 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec159ba0 msgr2=0x7f64ec0c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f22cf640 1 --2- 172.21.15.28:0/1323295641 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec159ba0 0x7f64ec0c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f22cf640 1 -- 172.21.15.28:0/1323295641 >> v1:172.21.15.122:6789/0 conn(0x7f64ec0cb0c0 legacy=0x7f64ec0cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.976+0000 7f64f22cf640 1 -- 172.21.15.28:0/1323295641 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec0cda30 con 0x7f64ec0cd570 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f22cf640 1 --2- 172.21.15.28:0/1323295641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec0cd570 0x7f64ec070580 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f64e40098a0 tx=0x7f64e401e700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe149a21fd0f705 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f0acc640 1 -- 172.21.15.28:0/1323295641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e40a8650 con 0x7f64ec0cd570 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f0acc640 1 -- 172.21.15.28:0/1323295641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e40a87b0 con 0x7f64ec0cd570 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f0acc640 1 -- 172.21.15.28:0/1323295641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e40a8b40 con 0x7f64ec0cd570 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 -- 172.21.15.28:0/1323295641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec0cd570 msgr2=0x7f64ec070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 --2- 172.21.15.28:0/1323295641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec0cd570 0x7f64ec070580 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f64e40098a0 tx=0x7f64e401e700 comp rx=0 tx=0).stop 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 -- 172.21.15.28:0/1323295641 shutdown_connections 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 --2- 172.21.15.28:0/1323295641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec0cd570 0x7f64ec070580 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 --2- 172.21.15.28:0/1323295641 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec159ba0 0x7f64ec0c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 -- 172.21.15.28:0/1323295641 >> 172.21.15.28:0/1323295641 conn(0x7f64ec0b0c50 msgr2=0x7f64ec0b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 -- 172.21.15.28:0/1323295641 shutdown_connections 2023-12-16T21:57:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 -- 172.21.15.28:0/1323295641 wait complete. 2023-12-16T21:57:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 Processor -- start 2023-12-16T21:57:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.980+0000 7f64f3d32640 1 -- start start 2023-12-16T21:57:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec0cb0c0 0x7f64ec0cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64ec0cd570 0x7f64ec157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec159ba0 0x7f64ec1f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec099b70 con 0x7f64ec159ba0 2023-12-16T21:57:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec157b80 con 0x7f64ec0cd570 2023-12-16T21:57:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec157e00 con 0x7f64ec0cb0c0 2023-12-16T21:57:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec159ba0 0x7f64ec1f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec159ba0 0x7f64ec1f87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57374/0 (socket says 172.21.15.28:57374) 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 -- 172.21.15.28:0/760679153 learned_addr learned my addr 172.21.15.28:0/760679153 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 -- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec0cb0c0 msgr2=0x7f64ec0cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f1ace640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec0cb0c0 0x7f64ec0cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f12cd640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64ec0cd570 0x7f64ec157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec0cb0c0 0x7f64ec0cf170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 -- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64ec0cd570 msgr2=0x7f64ec157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64ec0cd570 0x7f64ec157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 -- 172.21.15.28:0/760679153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40094d0 con 0x7f64ec159ba0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f12cd640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64ec0cd570 0x7f64ec157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f22cf640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec159ba0 0x7f64ec1f87a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f64e401c330 tx=0x7f64e401b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64e27fc640 1 -- 172.21.15.28:0/760679153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e40b3650 con 0x7f64ec159ba0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64e27fc640 1 -- 172.21.15.28:0/760679153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e40b37b0 con 0x7f64ec159ba0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec1580e0 con 0x7f64ec159ba0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64e27fc640 1 -- 172.21.15.28:0/760679153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e40b3b40 con 0x7f64ec159ba0 2023-12-16T21:57:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ec1586a0 con 0x7f64ec159ba0 2023-12-16T21:57:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec04e850 con 0x7f64ec159ba0 2023-12-16T21:57:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64e27fc640 1 -- 172.21.15.28:0/760679153 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f64e40b3ca0 con 0x7f64ec159ba0 2023-12-16T21:57:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64e27fc640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f64d0066470 0x7f64d0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64e27fc640 1 -- 172.21.15.28:0/760679153 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f64e411e140 con 0x7f64ec159ba0 2023-12-16T21:57:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.984+0000 7f64f1ace640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f64d0066470 0x7f64d0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.988+0000 7f64f1ace640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f64d0066470 0x7f64d0068930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0b3be0 tx=0x7f64e800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:40.988+0000 7f64e27fc640 1 -- 172.21.15.28:0/760679153 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f64e40eba50 con 0x7f64ec159ba0 2023-12-16T21:57:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.156+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f64ec0693d0 con 0x7f64ec159ba0 2023-12-16T21:57:41.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.460+0000 7f64e27fc640 1 -- 172.21.15.28:0/760679153 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f64e40ef070 con 0x7f64ec159ba0 2023-12-16T21:57:41.461 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2023-12-16T21:57:41.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f64d0066470 msgr2=0x7f64d0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:41.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f64d0066470 0x7f64d0068930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0b3be0 tx=0x7f64e800b040 comp rx=0 tx=0).stop 2023-12-16T21:57:41.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec159ba0 msgr2=0x7f64ec1f87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:41.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec159ba0 0x7f64ec1f87a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f64e401c330 tx=0x7f64e401b4a0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 shutdown_connections 2023-12-16T21:57:41.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f64d0066470 0x7f64d0068930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ec159ba0 0x7f64ec1f87a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64ec0cd570 0x7f64ec157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 --2- 172.21.15.28:0/760679153 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f64ec0cb0c0 0x7f64ec0cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 >> 172.21.15.28:0/760679153 conn(0x7f64ec0b0c50 msgr2=0x7f64ec0adf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:41.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 shutdown_connections 2023-12-16T21:57:41.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.464+0000 7f64f3d32640 1 -- 172.21.15.28:0/760679153 wait complete. 2023-12-16T21:57:41.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-16T21:57:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f5182ed6640 1 Processor -- start 2023-12-16T21:57:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f5182ed6640 1 -- start start 2023-12-16T21:57:41.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f5182ed6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f517c143d10 0x7f517c15f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:41.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f5182ed6640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f517c070480 0x7f517c15b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:41.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f5182ed6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517c1446f0 con 0x7f517c0b42d0 2023-12-16T21:57:41.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f5182ed6640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c071590 con 0x7f517c070480 2023-12-16T21:57:41.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f5182ed6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c070910 con 0x7f517c143d10 2023-12-16T21:57:41.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f517bfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f517c070480 0x7f517c15b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:41.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f517bfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f517c070480 0x7f517c15b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57388/0 (socket says 172.21.15.28:57388) 2023-12-16T21:57:41.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f517bfff640 1 -- 172.21.15.28:0/3771324267 learned_addr learned my addr 172.21.15.28:0/3771324267 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:41.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f5180c72640 1 --2- 172.21.15.28:0/3771324267 >> v2:172.21.15.122:3300/0 conn(0x7f517c143d10 0x7f517c15f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:41.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f517bfff640 1 -- 172.21.15.28:0/3771324267 >> v2:172.21.15.122:3300/0 conn(0x7f517c143d10 msgr2=0x7f517c15f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:41.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f517bfff640 1 --2- 172.21.15.28:0/3771324267 >> v2:172.21.15.122:3300/0 conn(0x7f517c143d10 0x7f517c15f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f517bfff640 1 -- 172.21.15.28:0/3771324267 >> v1:172.21.15.122:6789/0 conn(0x7f517c0b42d0 legacy=0x7f517c0719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:41.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.628+0000 7f517bfff640 1 -- 172.21.15.28:0/3771324267 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c070bf0 con 0x7f517c070480 2023-12-16T21:57:41.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f517bfff640 1 --2- 172.21.15.28:0/3771324267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c070480 0x7f517c15b5e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f517c06b860 tx=0x7f516401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54648cd7d927af7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f517b7fe640 1 -- 172.21.15.28:0/3771324267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51640ab640 con 0x7f517c070480 2023-12-16T21:57:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f517b7fe640 1 -- 172.21.15.28:0/3771324267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51640a7070 con 0x7f517c070480 2023-12-16T21:57:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f517b7fe640 1 -- 172.21.15.28:0/3771324267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51640ab9c0 con 0x7f517c070480 2023-12-16T21:57:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f5182ed6640 1 -- 172.21.15.28:0/3771324267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c070480 msgr2=0x7f517c15b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3771324267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c070480 0x7f517c15b5e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f517c06b860 tx=0x7f516401c440 comp rx=0 tx=0).stop 2023-12-16T21:57:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f5182ed6640 1 -- 172.21.15.28:0/3771324267 shutdown_connections 2023-12-16T21:57:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3771324267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c070480 0x7f517c15b5e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3771324267 >> v2:172.21.15.122:3300/0 conn(0x7f517c143d10 0x7f517c15f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f5182ed6640 1 -- 172.21.15.28:0/3771324267 >> 172.21.15.28:0/3771324267 conn(0x7f517c0b0cc0 msgr2=0x7f517c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:41.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f5182ed6640 1 -- 172.21.15.28:0/3771324267 shutdown_connections 2023-12-16T21:57:41.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.632+0000 7f5182ed6640 1 -- 172.21.15.28:0/3771324267 wait complete. 2023-12-16T21:57:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 Processor -- start 2023-12-16T21:57:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 -- start start 2023-12-16T21:57:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c0b42d0 0x7f517c15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f517c143d10 0x7f517c138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f517c154700 0x7f517c157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c139140 con 0x7f517c0b42d0 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c15cc80 con 0x7f517c143d10 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c15cf60 con 0x7f517c154700 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c0b42d0 0x7f517c15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c0b42d0 0x7f517c15af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57392/0 (socket says 172.21.15.28:57392) 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 -- 172.21.15.28:0/3295104675 learned_addr learned my addr 172.21.15.28:0/3295104675 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f517bfff640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f517c143d10 0x7f517c138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5181473640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f517c154700 0x7f517c157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:41.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 -- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f517c154700 msgr2=0x7f517c157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f517c154700 0x7f517c157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 -- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f517c143d10 msgr2=0x7f517c138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f517c143d10 0x7f517c138960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 -- 172.21.15.28:0/3295104675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640094d0 con 0x7f517c0b42d0 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5181473640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f517c154700 0x7f517c157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5180c72640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c0b42d0 0x7f517c15af00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5174007e40 tx=0x7f517400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f51797fa640 1 -- 172.21.15.28:0/3295104675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174014620 con 0x7f517c0b42d0 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f51797fa640 1 -- 172.21.15.28:0/3295104675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517400ee90 con 0x7f517c0b42d0 2023-12-16T21:57:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517c15d240 con 0x7f517c0b42d0 2023-12-16T21:57:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f51797fa640 1 -- 172.21.15.28:0/3295104675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51740149c0 con 0x7f517c0b42d0 2023-12-16T21:57:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c138ea0 con 0x7f517c0b42d0 2023-12-16T21:57:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c069440 con 0x7f517c0b42d0 2023-12-16T21:57:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.636+0000 7f51797fa640 1 -- 172.21.15.28:0/3295104675 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f5174014b20 con 0x7f517c0b42d0 2023-12-16T21:57:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.640+0000 7f51797fa640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f515c066470 0x7f515c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.640+0000 7f51797fa640 1 -- 172.21.15.28:0/3295104675 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f51740846d0 con 0x7f517c0b42d0 2023-12-16T21:57:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.640+0000 7f517bfff640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f515c066470 0x7f515c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.640+0000 7f517bfff640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f515c066470 0x7f515c068930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f51640af000 tx=0x7f51640a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.644+0000 7f51797fa640 1 -- 172.21.15.28:0/3295104675 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f5174051c90 con 0x7f517c0b42d0 2023-12-16T21:57:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:41.800+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f517c06e7a0 con 0x7f517c0b42d0 2023-12-16T21:57:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.472+0000 7f51797fa640 1 -- 172.21.15.28:0/3295104675 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v21) v1 ==== 170+0+0 (secure 0 0 0) 0x7f51740552b0 con 0x7f517c0b42d0 2023-12-16T21:57:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f515c066470 msgr2=0x7f515c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f515c066470 0x7f515c068930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f51640af000 tx=0x7f51640a7000 comp rx=0 tx=0).stop 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c0b42d0 msgr2=0x7f517c15af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c0b42d0 0x7f517c15af00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5174007e40 tx=0x7f517400e4b0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 shutdown_connections 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f515c066470 0x7f515c068930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f517c154700 0x7f517c157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f517c143d10 0x7f517c138960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 --2- 172.21.15.28:0/3295104675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c0b42d0 0x7f517c15af00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 >> 172.21.15.28:0/3295104675 conn(0x7f517c0b0cc0 msgr2=0x7f517c0cc340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 shutdown_connections 2023-12-16T21:57:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.476+0000 7f5182ed6640 1 -- 172.21.15.28:0/3295104675 wait complete. 2023-12-16T21:57:42.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-16T21:57:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.640+0000 7f42fdf57640 1 Processor -- start 2023-12-16T21:57:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.640+0000 7f42fdf57640 1 -- start start 2023-12-16T21:57:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42fdf57640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f42f8158ca0 0x7f42f815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42fdf57640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42f80cba10 0x7f42f815be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42fdf57640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f815b670 con 0x7f42f80b42d0 2023-12-16T21:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42fdf57640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f815f410 con 0x7f42f80cba10 2023-12-16T21:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42fdf57640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f815f6f0 con 0x7f42f8158ca0 2023-12-16T21:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f6ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42f80cba10 0x7f42f815be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f7fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f42f80b42d0 0x7f42f80703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50704/0 (socket says 172.21.15.28:50704) 2023-12-16T21:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f7fff640 1 -- 172.21.15.28:0/537618953 learned_addr learned my addr 172.21.15.28:0/537618953 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f77fe640 1 --2- 172.21.15.28:0/537618953 >> v2:172.21.15.122:3300/0 conn(0x7f42f8158ca0 0x7f42f815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f6ffd640 1 -- 172.21.15.28:0/537618953 >> v2:172.21.15.122:3300/0 conn(0x7f42f8158ca0 msgr2=0x7f42f815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f6ffd640 1 --2- 172.21.15.28:0/537618953 >> v2:172.21.15.122:3300/0 conn(0x7f42f8158ca0 0x7f42f815eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f6ffd640 1 -- 172.21.15.28:0/537618953 >> v1:172.21.15.28:6789/0 conn(0x7f42f80b42d0 legacy=0x7f42f80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f67fc640 1 -- 172.21.15.28:0/537618953 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469564895 0 0) 0x7f42f815b670 con 0x7f42f80b42d0 2023-12-16T21:57:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f6ffd640 1 -- 172.21.15.28:0/537618953 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f815b8e0 con 0x7f42f80cba10 2023-12-16T21:57:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.644+0000 7f42f6ffd640 1 --2- 172.21.15.28:0/537618953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815be00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f42f8067f70 tx=0x7f42ec01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2e572031fe529f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f67fc640 1 -- 172.21.15.28:0/537618953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0ab680 con 0x7f42f80cba10 2023-12-16T21:57:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f67fc640 1 -- 172.21.15.28:0/537618953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec0ab7e0 con 0x7f42f80cba10 2023-12-16T21:57:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f67fc640 1 -- 172.21.15.28:0/537618953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0abb70 con 0x7f42f80cba10 2023-12-16T21:57:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- 172.21.15.28:0/537618953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 msgr2=0x7f42f815be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 --2- 172.21.15.28:0/537618953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815be00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f42f8067f70 tx=0x7f42ec01c460 comp rx=0 tx=0).stop 2023-12-16T21:57:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- 172.21.15.28:0/537618953 shutdown_connections 2023-12-16T21:57:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 --2- 172.21.15.28:0/537618953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815be00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 --2- 172.21.15.28:0/537618953 >> v2:172.21.15.122:3300/0 conn(0x7f42f8158ca0 0x7f42f815eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- 172.21.15.28:0/537618953 >> 172.21.15.28:0/537618953 conn(0x7f42f80b0cc0 msgr2=0x7f42f80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- 172.21.15.28:0/537618953 shutdown_connections 2023-12-16T21:57:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- 172.21.15.28:0/537618953 wait complete. 2023-12-16T21:57:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 Processor -- start 2023-12-16T21:57:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- start start 2023-12-16T21:57:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f80b42d0 0x7f42f8158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42f8158ca0 0x7f42f814a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8157be0 con 0x7f42f80cba10 2023-12-16T21:57:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f815aa00 con 0x7f42f80b42d0 2023-12-16T21:57:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42fdf57640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8156fd0 con 0x7f42f8158ca0 2023-12-16T21:57:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57408/0 (socket says 172.21.15.28:57408) 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 -- 172.21.15.28:0/953732082 learned_addr learned my addr 172.21.15.28:0/953732082 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f7fff640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42f8158ca0 0x7f42f814a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 -- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42f8158ca0 msgr2=0x7f42f814a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f77fe640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f80b42d0 0x7f42f8158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42f8158ca0 0x7f42f814a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 -- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f80b42d0 msgr2=0x7f42f8158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f80b42d0 0x7f42f8158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 -- 172.21.15.28:0/953732082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec009420 con 0x7f42f80cba10 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f77fe640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f80b42d0 0x7f42f8158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42f6ffd640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815a4c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0ad040 tx=0x7f42ec01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42dffff640 1 -- 172.21.15.28:0/953732082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0b5810 con 0x7f42f80cba10 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42dffff640 1 -- 172.21.15.28:0/953732082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec0b5990 con 0x7f42f80cba10 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.648+0000 7f42dffff640 1 -- 172.21.15.28:0/953732082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0b5d40 con 0x7f42f80cba10 2023-12-16T21:57:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.652+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f81572b0 con 0x7f42f80cba10 2023-12-16T21:57:42.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.652+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f8157870 con 0x7f42f80cba10 2023-12-16T21:57:42.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.652+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f8157d50 con 0x7f42f80cba10 2023-12-16T21:57:42.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.652+0000 7f42dffff640 1 -- 172.21.15.28:0/953732082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f42ec0ca430 con 0x7f42f80cba10 2023-12-16T21:57:42.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.652+0000 7f42dffff640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42e4066470 0x7f42e4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:42.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.652+0000 7f42dffff640 1 -- 172.21.15.28:0/953732082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f42ec11ef90 con 0x7f42f80cba10 2023-12-16T21:57:42.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.652+0000 7f42f77fe640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42e4066470 0x7f42e4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:42.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.652+0000 7f42f77fe640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42e4066470 0x7f42e4068930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f42f814c8c0 tx=0x7f42e8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:42.656 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.656+0000 7f42dffff640 1 -- 172.21.15.28:0/953732082 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f42ec0ec570 con 0x7f42f80cba10 2023-12-16T21:57:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:42.848+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f42f80693d0 con 0x7f42f80cba10 2023-12-16T21:57:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.512+0000 7f42dffff640 1 -- 172.21.15.28:0/953732082 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f42ec0efb90 con 0x7f42f80cba10 2023-12-16T21:57:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2023-12-16T21:57:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42e4066470 msgr2=0x7f42e4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42e4066470 0x7f42e4068930 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f42f814c8c0 tx=0x7f42e8008040 comp rx=0 tx=0).stop 2023-12-16T21:57:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 msgr2=0x7f42f815a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815a4c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0ad040 tx=0x7f42ec01aca0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 shutdown_connections 2023-12-16T21:57:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42e4066470 0x7f42e4068930 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42f8158ca0 0x7f42f814a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42f80cba10 0x7f42f815a4c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 --2- 172.21.15.28:0/953732082 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42f80b42d0 0x7f42f8158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 >> 172.21.15.28:0/953732082 conn(0x7f42f80b0cc0 msgr2=0x7f42f80b10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:43.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 shutdown_connections 2023-12-16T21:57:43.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.516+0000 7f42fdf57640 1 -- 172.21.15.28:0/953732082 wait complete. 2023-12-16T21:57:43.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae51594640 1 Processor -- start 2023-12-16T21:57:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae51594640 1 -- start start 2023-12-16T21:57:43.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae51594640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0b35b0 0x7fae4c0d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae51594640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae4c0cfe90 0x7fae4c15f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae51594640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4c0d3880 con 0x7fae4c0b3c70 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae51594640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0d4090 con 0x7fae4c0cfe90 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae51594640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0cdc80 con 0x7fae4c0b35b0 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae4b7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae4c0cfe90 0x7fae4c15f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae4b7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae4c0cfe90 0x7fae4c15f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57420/0 (socket says 172.21.15.28:57420) 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae4b7fe640 1 -- 172.21.15.28:0/2993456738 learned_addr learned my addr 172.21.15.28:0/2993456738 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae4affd640 1 --2- 172.21.15.28:0/2993456738 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0b35b0 0x7fae4c0d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae4b7fe640 1 -- 172.21.15.28:0/2993456738 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0b35b0 msgr2=0x7fae4c0d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae4b7fe640 1 --2- 172.21.15.28:0/2993456738 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0b35b0 0x7fae4c0d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae4b7fe640 1 -- 172.21.15.28:0/2993456738 >> v1:172.21.15.122:6789/0 conn(0x7fae4c0b3c70 legacy=0x7fae4c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.664+0000 7fae4b7fe640 1 -- 172.21.15.28:0/2993456738 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0d3ad0 con 0x7fae4c0cfe90 2023-12-16T21:57:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae4b7fe640 1 --2- 172.21.15.28:0/2993456738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0cfe90 0x7fae4c15f060 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fae44009e50 tx=0x7fae4401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8958e707b1edf718 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae49ffb640 1 -- 172.21.15.28:0/2993456738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae440a5850 con 0x7fae4c0cfe90 2023-12-16T21:57:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae49ffb640 1 -- 172.21.15.28:0/2993456738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae440a59b0 con 0x7fae4c0cfe90 2023-12-16T21:57:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae49ffb640 1 -- 172.21.15.28:0/2993456738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae440a5d40 con 0x7fae4c0cfe90 2023-12-16T21:57:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- 172.21.15.28:0/2993456738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0cfe90 msgr2=0x7fae4c15f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 --2- 172.21.15.28:0/2993456738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0cfe90 0x7fae4c15f060 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fae44009e50 tx=0x7fae4401c9d0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- 172.21.15.28:0/2993456738 shutdown_connections 2023-12-16T21:57:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 --2- 172.21.15.28:0/2993456738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0cfe90 0x7fae4c15f060 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 --2- 172.21.15.28:0/2993456738 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0b35b0 0x7fae4c0d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- 172.21.15.28:0/2993456738 >> 172.21.15.28:0/2993456738 conn(0x7fae4c09c4b0 msgr2=0x7fae4c0d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- 172.21.15.28:0/2993456738 shutdown_connections 2023-12-16T21:57:43.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- 172.21.15.28:0/2993456738 wait complete. 2023-12-16T21:57:43.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 Processor -- start 2023-12-16T21:57:43.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- start start 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4c0b35b0 0x7fae4c06c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0b3c70 0x7fae4c0cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0cfe90 0x7fae4c15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c1537e0 con 0x7fae4c0b3c70 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c15df10 con 0x7fae4c0b35b0 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae51594640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c15e1f0 con 0x7fae4c0cfe90 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae4a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0b3c70 0x7fae4c0cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae4a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0b3c70 0x7fae4c0cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57432/0 (socket says 172.21.15.28:57432) 2023-12-16T21:57:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.668+0000 7fae4a7fc640 1 -- 172.21.15.28:0/891009553 learned_addr learned my addr 172.21.15.28:0/891009553 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4b7fe640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0cfe90 0x7fae4c15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4affd640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4c0b35b0 0x7fae4c06c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4a7fc640 1 -- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0cfe90 msgr2=0x7fae4c15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4a7fc640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0cfe90 0x7fae4c15d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4a7fc640 1 -- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4c0b35b0 msgr2=0x7fae4c06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4a7fc640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4c0b35b0 0x7fae4c06c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4a7fc640 1 -- 172.21.15.28:0/891009553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae44009b30 con 0x7fae4c0b3c70 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4affd640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4c0b35b0 0x7fae4c06c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4a7fc640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0b3c70 0x7fae4c0cb890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fae4000c450 tx=0x7fae4000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae337fe640 1 -- 172.21.15.28:0/891009553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae40012830 con 0x7fae4c0b3c70 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae337fe640 1 -- 172.21.15.28:0/891009553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae400129b0 con 0x7fae4c0b3c70 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c153470 con 0x7fae4c0b3c70 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae337fe640 1 -- 172.21.15.28:0/891009553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae40012d60 con 0x7fae4c0b3c70 2023-12-16T21:57:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c06ce00 con 0x7fae4c0b3c70 2023-12-16T21:57:43.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c1532d0 con 0x7fae4c0b3c70 2023-12-16T21:57:43.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae337fe640 1 -- 172.21.15.28:0/891009553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fae4001c460 con 0x7fae4c0b3c70 2023-12-16T21:57:43.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae337fe640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae38066470 0x7fae38068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:43.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae337fe640 1 -- 172.21.15.28:0/891009553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae40083ce0 con 0x7fae4c0b3c70 2023-12-16T21:57:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4affd640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae38066470 0x7fae38068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.672+0000 7fae4affd640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae38066470 0x7fae38068930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0c9ed0 tx=0x7fae3c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:43.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.676+0000 7fae337fe640 1 -- 172.21.15.28:0/891009553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fae4c1532d0 con 0x7fae4c0b3c70 2023-12-16T21:57:43.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:57:43.777 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:57:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.836+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae4c1532d0 con 0x7fae4c0b3c70 2023-12-16T21:57:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.836+0000 7fae337fe640 1 -- 172.21.15.28:0/891009553 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16506 (secure 0 0 0) 0x7fae4c1532d0 con 0x7fae4c0b3c70 2023-12-16T21:57:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:43.838 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:42.850229+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae38066470 msgr2=0x7fae38068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae38066470 0x7fae38068930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0c9ed0 tx=0x7fae3c008040 comp rx=0 tx=0).stop 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0b3c70 msgr2=0x7fae4c0cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0b3c70 0x7fae4c0cb890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fae4000c450 tx=0x7fae4000c910 comp rx=0 tx=0).stop 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 shutdown_connections 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae38066470 0x7fae38068930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae4c0cfe90 0x7fae4c15d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae4c0b3c70 0x7fae4c0cb890 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 --2- 172.21.15.28:0/891009553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae4c0b35b0 0x7fae4c06c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 >> 172.21.15.28:0/891009553 conn(0x7fae4c09c4b0 msgr2=0x7fae4c0b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 shutdown_connections 2023-12-16T21:57:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:43.840+0000 7fae51594640 1 -- 172.21.15.28:0/891009553 wait complete. 2023-12-16T21:57:43.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e2d713640 1 Processor -- start 2023-12-16T21:57:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e2d713640 1 -- start start 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e2d713640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e2806c2c0 0x7f8e2806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e2d713640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e28072480 con 0x7f8e28072010 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e2d713640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e28071470 con 0x7f8e28071b90 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e2d713640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e280726e0 con 0x7f8e2806c2c0 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e26ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e2806c2c0 0x7f8e2806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e277fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8e28072010 0x7f8e2815f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50710/0 (socket says 172.21.15.28:50710) 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e26ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e2806c2c0 0x7f8e2806c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57444/0 (socket says 172.21.15.28:57444) 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e277fe640 1 -- 172.21.15.28:0/485640783 learned_addr learned my addr 172.21.15.28:0/485640783 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e26ffd640 1 -- 172.21.15.28:0/485640783 >> v1:172.21.15.122:6789/0 conn(0x7f8e28071b90 legacy=0x7f8e2815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e26ffd640 1 -- 172.21.15.28:0/485640783 >> v1:172.21.15.28:6789/0 conn(0x7f8e28072010 legacy=0x7f8e2815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e25ffb640 1 -- 172.21.15.28:0/485640783 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636685859 0 0) 0x7f8e28072480 con 0x7f8e28072010 2023-12-16T21:57:44.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.016+0000 7f8e26ffd640 1 -- 172.21.15.28:0/485640783 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2815fa70 con 0x7f8e2806c2c0 2023-12-16T21:57:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e26ffd640 1 --2- 172.21.15.28:0/485640783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2806c2c0 0x7f8e2806c6a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e180097c0 tx=0x7f8e1801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fee2c309b7848c2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e25ffb640 1 -- 172.21.15.28:0/485640783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180ac410 con 0x7f8e2806c2c0 2023-12-16T21:57:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e25ffb640 1 -- 172.21.15.28:0/485640783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e180ac570 con 0x7f8e2806c2c0 2023-12-16T21:57:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e25ffb640 1 -- 172.21.15.28:0/485640783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e180ac900 con 0x7f8e2806c2c0 2023-12-16T21:57:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- 172.21.15.28:0/485640783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2806c2c0 msgr2=0x7f8e2806c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 --2- 172.21.15.28:0/485640783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2806c2c0 0x7f8e2806c6a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e180097c0 tx=0x7f8e1801c3b0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- 172.21.15.28:0/485640783 shutdown_connections 2023-12-16T21:57:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 --2- 172.21.15.28:0/485640783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e2806c2c0 0x7f8e2806c6a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- 172.21.15.28:0/485640783 >> 172.21.15.28:0/485640783 conn(0x7f8e2806a530 msgr2=0x7f8e2806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- 172.21.15.28:0/485640783 shutdown_connections 2023-12-16T21:57:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- 172.21.15.28:0/485640783 wait complete. 2023-12-16T21:57:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 Processor -- start 2023-12-16T21:57:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- start start 2023-12-16T21:57:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e2806c2c0 0x7f8e281f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e28071b90 0x7f8e28142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8e28072010 0x7f8e28155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e28156910 con 0x7f8e28071b90 2023-12-16T21:57:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e281f3e60 con 0x7f8e2806c2c0 2023-12-16T21:57:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e2d713640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e28156350 con 0x7f8e28072010 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e267fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e28071b90 0x7f8e28142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e267fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e28071b90 0x7f8e28142f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57450/0 (socket says 172.21.15.28:57450) 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e267fc640 1 -- 172.21.15.28:0/3337528671 learned_addr learned my addr 172.21.15.28:0/3337528671 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e26ffd640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e2806c2c0 0x7f8e281f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e277fe640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8e28072010 0x7f8e28155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e267fc640 1 -- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8e28072010 msgr2=0x7f8e28155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e267fc640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8e28072010 0x7f8e28155a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e267fc640 1 -- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e2806c2c0 msgr2=0x7f8e281f3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e267fc640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e2806c2c0 0x7f8e281f3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e267fc640 1 -- 172.21.15.28:0/3337528671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e18009420 con 0x7f8e28071b90 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.020+0000 7f8e26ffd640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e2806c2c0 0x7f8e281f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e267fc640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e28071b90 0x7f8e28142f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000b410 tx=0x7f8e1000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e1f7fe640 1 -- 172.21.15.28:0/3337528671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10004020 con 0x7f8e28071b90 2023-12-16T21:57:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e1f7fe640 1 -- 172.21.15.28:0/3337528671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e10004180 con 0x7f8e28071b90 2023-12-16T21:57:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e28156630 con 0x7f8e28071b90 2023-12-16T21:57:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e1f7fe640 1 -- 172.21.15.28:0/3337528671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e10011630 con 0x7f8e28071b90 2023-12-16T21:57:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e281577e0 con 0x7f8e28071b90 2023-12-16T21:57:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2815baa0 con 0x7f8e28071b90 2023-12-16T21:57:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e1f7fe640 1 -- 172.21.15.28:0/3337528671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f8e10011790 con 0x7f8e28071b90 2023-12-16T21:57:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e1f7fe640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8e08066470 0x7f8e08068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e1f7fe640 1 -- 172.21.15.28:0/3337528671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e10083b50 con 0x7f8e28071b90 2023-12-16T21:57:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e26ffd640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8e08066470 0x7f8e08068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.024+0000 7f8e26ffd640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8e08066470 0x7f8e08068930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8e180ae000 tx=0x7f8e18008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.028+0000 7f8e1f7fe640 1 -- 172.21.15.28:0/3337528671 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f8e10051060 con 0x7f8e28071b90 2023-12-16T21:57:44.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.184+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e28069440 con 0x7f8e28071b90 2023-12-16T21:57:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.188+0000 7f8e1f7fe640 1 -- 172.21.15.28:0/3337528671 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16506 (secure 0 0 0) 0x7f8e10054680 con 0x7f8e28071b90 2023-12-16T21:57:44.188 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:44.189 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:42.850229+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8e08066470 msgr2=0x7f8e08068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8e08066470 0x7f8e08068930 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8e180ae000 tx=0x7f8e18008040 comp rx=0 tx=0).stop 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e28071b90 msgr2=0x7f8e28142f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e28071b90 0x7f8e28142f50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000b410 tx=0x7f8e1000b8d0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 shutdown_connections 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8e08066470 0x7f8e08068930 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8e28072010 0x7f8e28155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e28071b90 0x7f8e28142f50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 --2- 172.21.15.28:0/3337528671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e2806c2c0 0x7f8e281f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 >> 172.21.15.28:0/3337528671 conn(0x7f8e2806a530 msgr2=0x7f8e280cc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 shutdown_connections 2023-12-16T21:57:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.192+0000 7f8e2d713640 1 -- 172.21.15.28:0/3337528671 wait complete. 2023-12-16T21:57:44.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-16T21:57:44.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f0532be1640 1 Processor -- start 2023-12-16T21:57:44.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f0532be1640 1 -- start start 2023-12-16T21:57:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f0532be1640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f052c071d90 0x7f052c072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f0532be1640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c13d2b0 con 0x7f052c15adb0 2023-12-16T21:57:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f0532be1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052c0cdd70 con 0x7f052c13f5f0 2023-12-16T21:57:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f0532be1640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f052c140520 con 0x7f052c071d90 2023-12-16T21:57:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f053097d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f052c071d90 0x7f052c072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f052bfff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f052c15adb0 0x7f052c13d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50722/0 (socket says 172.21.15.28:50722) 2023-12-16T21:57:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f052bfff640 1 -- 172.21.15.28:0/848602312 learned_addr learned my addr 172.21.15.28:0/848602312 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:44.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f053097d640 1 -- 172.21.15.28:0/848602312 >> v1:172.21.15.122:6789/0 conn(0x7f052c13f5f0 legacy=0x7f052c0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f053097d640 1 -- 172.21.15.28:0/848602312 >> v1:172.21.15.28:6789/0 conn(0x7f052c15adb0 legacy=0x7f052c13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f052b7fe640 1 -- 172.21.15.28:0/848602312 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690138189 0 0) 0x7f052c13d2b0 con 0x7f052c15adb0 2023-12-16T21:57:44.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.332+0000 7f053097d640 1 -- 172.21.15.28:0/848602312 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c0726b0 con 0x7f052c071d90 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f053097d640 1 --2- 172.21.15.28:0/848602312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c071d90 0x7f052c072170 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f051c0097c0 tx=0x7f051c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672f14346af466c3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f052b7fe640 1 -- 172.21.15.28:0/848602312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c0ab600 con 0x7f052c071d90 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f052b7fe640 1 -- 172.21.15.28:0/848602312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051c0a7050 con 0x7f052c071d90 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f052b7fe640 1 -- 172.21.15.28:0/848602312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c0ab980 con 0x7f052c071d90 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 -- 172.21.15.28:0/848602312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c071d90 msgr2=0x7f052c072170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 --2- 172.21.15.28:0/848602312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c071d90 0x7f052c072170 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f051c0097c0 tx=0x7f051c01c480 comp rx=0 tx=0).stop 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 -- 172.21.15.28:0/848602312 shutdown_connections 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 --2- 172.21.15.28:0/848602312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c071d90 0x7f052c072170 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 -- 172.21.15.28:0/848602312 >> 172.21.15.28:0/848602312 conn(0x7f052c0b0cc0 msgr2=0x7f052c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 -- 172.21.15.28:0/848602312 shutdown_connections 2023-12-16T21:57:44.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 -- 172.21.15.28:0/848602312 wait complete. 2023-12-16T21:57:44.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 Processor -- start 2023-12-16T21:57:44.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.336+0000 7f0532be1640 1 -- start start 2023-12-16T21:57:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f052c071d90 0x7f052c15bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f052c13f5f0 0x7f052c138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c15adb0 0x7f052c13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c13c670 con 0x7f052c15adb0 2023-12-16T21:57:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c0ae280 con 0x7f052c071d90 2023-12-16T21:57:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052c15b9b0 con 0x7f052c13f5f0 2023-12-16T21:57:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c15adb0 0x7f052c13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c15adb0 0x7f052c13a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57472/0 (socket says 172.21.15.28:57472) 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 -- 172.21.15.28:0/1232144978 learned_addr learned my addr 172.21.15.28:0/1232144978 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053097d640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f052c071d90 0x7f052c15bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f052bfff640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f052c13f5f0 0x7f052c138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 -- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f052c13f5f0 msgr2=0x7f052c138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f052c13f5f0 0x7f052c138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 -- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f052c071d90 msgr2=0x7f052c15bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f052c071d90 0x7f052c15bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 -- 172.21.15.28:0/1232144978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c009420 con 0x7f052c15adb0 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f053117e640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c15adb0 0x7f052c13a7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f052400c770 tx=0x7f052400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f05297fa640 1 -- 172.21.15.28:0/1232144978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0524003b80 con 0x7f052c15adb0 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c15b6d0 con 0x7f052c15adb0 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f05297fa640 1 -- 172.21.15.28:0/1232144978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0524003ce0 con 0x7f052c15adb0 2023-12-16T21:57:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f05297fa640 1 -- 172.21.15.28:0/1232144978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052400aa90 con 0x7f052c15adb0 2023-12-16T21:57:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c0adea0 con 0x7f052c15adb0 2023-12-16T21:57:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.340+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c13c7b0 con 0x7f052c15adb0 2023-12-16T21:57:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.344+0000 7f05297fa640 1 -- 172.21.15.28:0/1232144978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f052400abf0 con 0x7f052c15adb0 2023-12-16T21:57:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.344+0000 7f05297fa640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0518066470 0x7f0518068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.344+0000 7f05297fa640 1 -- 172.21.15.28:0/1232144978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0524084570 con 0x7f052c15adb0 2023-12-16T21:57:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.344+0000 7f053097d640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0518066470 0x7f0518068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:44.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.344+0000 7f053097d640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0518066470 0x7f0518068930 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f051c0ad000 tx=0x7f051c0a6000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.344+0000 7f05297fa640 1 -- 172.21.15.28:0/1232144978 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f0524051a80 con 0x7f052c15adb0 2023-12-16T21:57:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:44.540+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f052c13c7b0 con 0x7f052c15adb0 2023-12-16T21:57:45.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.528+0000 7f05297fa640 1 -- 172.21.15.28:0/1232144978 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f052c13c7b0 con 0x7f052c15adb0 2023-12-16T21:57:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0518066470 msgr2=0x7f0518068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0518066470 0x7f0518068930 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f051c0ad000 tx=0x7f051c0a6000 comp rx=0 tx=0).stop 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c15adb0 msgr2=0x7f052c13a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c15adb0 0x7f052c13a7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f052400c770 tx=0x7f052400cc30 comp rx=0 tx=0).stop 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 shutdown_connections 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0518066470 0x7f0518068930 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f052c15adb0 0x7f052c13a7d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f052c13f5f0 0x7f052c138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 --2- 172.21.15.28:0/1232144978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f052c071d90 0x7f052c15bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 >> 172.21.15.28:0/1232144978 conn(0x7f052c0b0cc0 msgr2=0x7f052c136120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 shutdown_connections 2023-12-16T21:57:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.532+0000 7f0532be1640 1 -- 172.21.15.28:0/1232144978 wait complete. 2023-12-16T21:57:45.554 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-12-16T21:57:45.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-16T21:57:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cfdb1640 1 Processor -- start 2023-12-16T21:57:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cfdb1640 1 -- start start 2023-12-16T21:57:45.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cfdb1640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff0c815adb0 0x7ff0c813d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:45.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cfdb1640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c813d2b0 con 0x7ff0c8071d90 2023-12-16T21:57:45.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cfdb1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c80cdd70 con 0x7ff0c813f5f0 2023-12-16T21:57:45.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cfdb1640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c8140520 con 0x7ff0c815adb0 2023-12-16T21:57:45.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cd34c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff0c815adb0 0x7ff0c813d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:45.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cdb4d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff0c8071d90 0x7ff0c8072170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50732/0 (socket says 172.21.15.28:50732) 2023-12-16T21:57:45.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cdb4d640 1 -- 172.21.15.28:0/2819361170 learned_addr learned my addr 172.21.15.28:0/2819361170 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:45.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cd34c640 1 -- 172.21.15.28:0/2819361170 >> v1:172.21.15.122:6789/0 conn(0x7ff0c813f5f0 legacy=0x7ff0c80ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:45.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cd34c640 1 -- 172.21.15.28:0/2819361170 >> v1:172.21.15.28:6789/0 conn(0x7ff0c8071d90 legacy=0x7ff0c8072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:45.689 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.688+0000 7ff0cd34c640 1 -- 172.21.15.28:0/2819361170 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c813ff60 con 0x7ff0c815adb0 2023-12-16T21:57:45.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 --2- 172.21.15.28:0/2819361170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c815adb0 0x7ff0c813d560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80097c0 tx=0x7ff0b801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93acb3a78fdd07f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:45.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0ccb4b640 1 -- 172.21.15.28:0/2819361170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b80ab600 con 0x7ff0c815adb0 2023-12-16T21:57:45.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0ccb4b640 1 -- 172.21.15.28:0/2819361170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b80a7050 con 0x7ff0c815adb0 2023-12-16T21:57:45.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0ccb4b640 1 -- 172.21.15.28:0/2819361170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b80ab980 con 0x7ff0c815adb0 2023-12-16T21:57:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/2819361170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c815adb0 msgr2=0x7ff0c813d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 --2- 172.21.15.28:0/2819361170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c815adb0 0x7ff0c813d560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80097c0 tx=0x7ff0b801c440 comp rx=0 tx=0).stop 2023-12-16T21:57:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/2819361170 shutdown_connections 2023-12-16T21:57:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 --2- 172.21.15.28:0/2819361170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c815adb0 0x7ff0c813d560 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/2819361170 >> 172.21.15.28:0/2819361170 conn(0x7ff0c80b0cc0 msgr2=0x7ff0c80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/2819361170 shutdown_connections 2023-12-16T21:57:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/2819361170 wait complete. 2023-12-16T21:57:45.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 Processor -- start 2023-12-16T21:57:45.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- start start 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0c8071d90 0x7ff0c8071440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c813f5f0 0x7ff0c8133000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff0c815adb0 0x7ff0c814ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c8133540 con 0x7ff0c813f5f0 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c813e640 con 0x7ff0c8071d90 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cfdb1640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c81347c0 con 0x7ff0c815adb0 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c813f5f0 0x7ff0c8133000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c813f5f0 0x7ff0c8133000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57498/0 (socket says 172.21.15.28:57498) 2023-12-16T21:57:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 -- 172.21.15.28:0/1949777750 learned_addr learned my addr 172.21.15.28:0/1949777750 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cdb4d640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0c8071d90 0x7ff0c8071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 -- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff0c815adb0 msgr2=0x7ff0c814ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0ce34e640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff0c815adb0 0x7ff0c814ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff0c815adb0 0x7ff0c814ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 -- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0c8071d90 msgr2=0x7ff0c8071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0c8071d90 0x7ff0c8071440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 -- 172.21.15.28:0/1949777750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b8009420 con 0x7ff0c813f5f0 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cdb4d640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0c8071d90 0x7ff0c8071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0ce34e640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff0c815adb0 0x7ff0c814ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.692+0000 7ff0cd34c640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c813f5f0 0x7ff0c8133000 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8019050 tx=0x7ff0b80191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0be7fc640 1 -- 172.21.15.28:0/1949777750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b80bb070 con 0x7ff0c813f5f0 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0be7fc640 1 -- 172.21.15.28:0/1949777750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b80abe50 con 0x7ff0c813f5f0 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c8139a20 con 0x7ff0c813f5f0 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0be7fc640 1 -- 172.21.15.28:0/1949777750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b80b4640 con 0x7ff0c813f5f0 2023-12-16T21:57:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c8134450 con 0x7ff0c813f5f0 2023-12-16T21:57:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c80ced20 con 0x7ff0c813f5f0 2023-12-16T21:57:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0be7fc640 1 -- 172.21.15.28:0/1949777750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7ff0b80b2040 con 0x7ff0c813f5f0 2023-12-16T21:57:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0be7fc640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff0ac066470 0x7ff0ac068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0be7fc640 1 -- 172.21.15.28:0/1949777750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0b811d460 con 0x7ff0c813f5f0 2023-12-16T21:57:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0cdb4d640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff0ac066470 0x7ff0ac068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.696+0000 7ff0cdb4d640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff0ac066470 0x7ff0ac068930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4007fd0 tx=0x7ff0c4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:45.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.700+0000 7ff0be7fc640 1 -- 172.21.15.28:0/1949777750 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7ff0b80ea970 con 0x7ff0c813f5f0 2023-12-16T21:57:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:45.888+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff0c8134930 con 0x7ff0c813f5f0 2023-12-16T21:57:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.544+0000 7ff0be7fc640 1 -- 172.21.15.28:0/1949777750 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff0b80edf90 con 0x7ff0c813f5f0 2023-12-16T21:57:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff0ac066470 msgr2=0x7ff0ac068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff0ac066470 0x7ff0ac068930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4007fd0 tx=0x7ff0c4009040 comp rx=0 tx=0).stop 2023-12-16T21:57:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c813f5f0 msgr2=0x7ff0c8133000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c813f5f0 0x7ff0c8133000 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8019050 tx=0x7ff0b80191e0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 shutdown_connections 2023-12-16T21:57:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff0ac066470 0x7ff0ac068930 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff0c815adb0 0x7ff0c814ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c813f5f0 0x7ff0c8133000 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 --2- 172.21.15.28:0/1949777750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0c8071d90 0x7ff0c8071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 >> 172.21.15.28:0/1949777750 conn(0x7ff0c80b0cc0 msgr2=0x7ff0c80af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 shutdown_connections 2023-12-16T21:57:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.548+0000 7ff0cfdb1640 1 -- 172.21.15.28:0/1949777750 wait complete. 2023-12-16T21:57:46.569 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-16T21:57:46.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:57:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae3881640 1 Processor -- start 2023-12-16T21:57:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae3881640 1 -- start start 2023-12-16T21:57:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae3881640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0cfe90 0x7f7adc0d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae3881640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7adc0b3c70 0x7f7adc15f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae3881640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7adc0d3880 con 0x7f7adc0b35b0 2023-12-16T21:57:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae3881640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7adc0d4090 con 0x7f7adc0b3c70 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae3881640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0cdc80 con 0x7f7adc0cfe90 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae161d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7adc0b35b0 0x7f7adc0cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50736/0 (socket says 172.21.15.28:50736) 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae1e1e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7adc0b3c70 0x7f7adc15f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae161d640 1 -- 172.21.15.28:0/2019333292 learned_addr learned my addr 172.21.15.28:0/2019333292 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae0e1c640 1 --2- 172.21.15.28:0/2019333292 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0cfe90 0x7f7adc0d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae1e1e640 1 -- 172.21.15.28:0/2019333292 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0cfe90 msgr2=0x7f7adc0d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae1e1e640 1 --2- 172.21.15.28:0/2019333292 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0cfe90 0x7f7adc0d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae1e1e640 1 -- 172.21.15.28:0/2019333292 >> v1:172.21.15.28:6789/0 conn(0x7f7adc0b35b0 legacy=0x7f7adc0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ae1e1e640 1 -- 172.21.15.28:0/2019333292 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc0d3ad0 con 0x7f7adc0b3c70 2023-12-16T21:57:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.704+0000 7f7ad3fff640 1 -- 172.21.15.28:0/2019333292 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735477491 0 0) 0x7f7adc0d3880 con 0x7f7adc0b35b0 2023-12-16T21:57:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae1e1e640 1 --2- 172.21.15.28:0/2019333292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0b3c70 0x7f7adc15f060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009d90 tx=0x7f7ad401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c80727d57b8a5d9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ad3fff640 1 -- 172.21.15.28:0/2019333292 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad40a5980 con 0x7f7adc0b3c70 2023-12-16T21:57:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ad3fff640 1 -- 172.21.15.28:0/2019333292 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad40a5ae0 con 0x7f7adc0b3c70 2023-12-16T21:57:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ad3fff640 1 -- 172.21.15.28:0/2019333292 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad40a5e70 con 0x7f7adc0b3c70 2023-12-16T21:57:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- 172.21.15.28:0/2019333292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0b3c70 msgr2=0x7f7adc15f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 --2- 172.21.15.28:0/2019333292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0b3c70 0x7f7adc15f060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009d90 tx=0x7f7ad401c8c0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- 172.21.15.28:0/2019333292 shutdown_connections 2023-12-16T21:57:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 --2- 172.21.15.28:0/2019333292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0b3c70 0x7f7adc15f060 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 --2- 172.21.15.28:0/2019333292 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0cfe90 0x7f7adc0d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- 172.21.15.28:0/2019333292 >> 172.21.15.28:0/2019333292 conn(0x7f7adc09c4b0 msgr2=0x7f7adc0d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- 172.21.15.28:0/2019333292 shutdown_connections 2023-12-16T21:57:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- 172.21.15.28:0/2019333292 wait complete. 2023-12-16T21:57:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 Processor -- start 2023-12-16T21:57:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- start start 2023-12-16T21:57:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0b35b0 0x7f7adc15b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7adc0b3c70 0x7f7adc0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0cfe90 0x7f7adc0a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc15b0b0 con 0x7f7adc0cfe90 2023-12-16T21:57:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0cc720 con 0x7f7adc0b3c70 2023-12-16T21:57:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae3881640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc0cd1c0 con 0x7f7adc0b35b0 2023-12-16T21:57:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae1e1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0cfe90 0x7f7adc0a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae1e1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0cfe90 0x7f7adc0a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57514/0 (socket says 172.21.15.28:57514) 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae1e1e640 1 -- 172.21.15.28:0/862757302 learned_addr learned my addr 172.21.15.28:0/862757302 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.708+0000 7f7ae161d640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0b35b0 0x7f7adc15b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae0e1c640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7adc0b3c70 0x7f7adc0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae1e1e640 1 -- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0b35b0 msgr2=0x7f7adc15b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae1e1e640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0b35b0 0x7f7adc15b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae1e1e640 1 -- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7adc0b3c70 msgr2=0x7f7adc0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae1e1e640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7adc0b3c70 0x7f7adc0cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae1e1e640 1 -- 172.21.15.28:0/862757302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad4009a70 con 0x7f7adc0cfe90 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae161d640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0b35b0 0x7f7adc15b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae1e1e640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0cfe90 0x7f7adc0a03e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ad401af70 tx=0x7f7ad4019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ad1ffb640 1 -- 172.21.15.28:0/862757302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad40ad850 con 0x7f7adc0cfe90 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ad1ffb640 1 -- 172.21.15.28:0/862757302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad40a5970 con 0x7f7adc0cfe90 2023-12-16T21:57:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ad1ffb640 1 -- 172.21.15.28:0/862757302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad40adbb0 con 0x7f7adc0cfe90 2023-12-16T21:57:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc0cc920 con 0x7f7adc0cfe90 2023-12-16T21:57:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc0ccee0 con 0x7f7adc0cfe90 2023-12-16T21:57:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc069440 con 0x7f7adc0cfe90 2023-12-16T21:57:46.717 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ad1ffb640 1 -- 172.21.15.28:0/862757302 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7ad40b4050 con 0x7f7adc0cfe90 2023-12-16T21:57:46.717 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ad1ffb640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ac0066470 0x7f7ac0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:46.718 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ad1ffb640 1 -- 172.21.15.28:0/862757302 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ad411ec70 con 0x7f7adc0cfe90 2023-12-16T21:57:46.718 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae161d640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ac0066470 0x7f7ac0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:46.718 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.712+0000 7f7ae161d640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ac0066470 0x7f7ac0068930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8000f70 tx=0x7f7ad8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:46.718 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.716+0000 7f7ad1ffb640 1 -- 172.21.15.28:0/862757302 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7ad40ec180 con 0x7f7adc0cfe90 2023-12-16T21:57:46.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.904+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7adc0d0720 con 0x7f7adc0cfe90 2023-12-16T21:57:46.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ad1ffb640 1 -- 172.21.15.28:0/862757302 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4932 (secure 0 0 0) 0x7f7ad40b2090 con 0x7f7adc0cfe90 2023-12-16T21:57:46.908 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:46.908 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:57:42.850798+0000","modified":"2023-12-16T21:57:45.892479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:57:46.908 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2023-12-16T21:57:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ac0066470 msgr2=0x7f7ac0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ac0066470 0x7f7ac0068930 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8000f70 tx=0x7f7ad8009040 comp rx=0 tx=0).stop 2023-12-16T21:57:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0cfe90 msgr2=0x7f7adc0a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0cfe90 0x7f7adc0a03e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ad401af70 tx=0x7f7ad4019660 comp rx=0 tx=0).stop 2023-12-16T21:57:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 shutdown_connections 2023-12-16T21:57:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ac0066470 0x7f7ac0068930 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7adc0cfe90 0x7f7adc0a03e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7adc0b3c70 0x7f7adc0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.911 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 --2- 172.21.15.28:0/862757302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7adc0b35b0 0x7f7adc15b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:46.911 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 >> 172.21.15.28:0/862757302 conn(0x7f7adc09c4b0 msgr2=0x7f7adc0b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:46.911 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 shutdown_connections 2023-12-16T21:57:46.911 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:46.908+0000 7f7ae3881640 1 -- 172.21.15.28:0/862757302 wait complete. 2023-12-16T21:57:46.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05b606b640 1 Processor -- start 2023-12-16T21:57:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05b606b640 1 -- start start 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05b606b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05b00b2ed0 0x7f05b0071430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05b606b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f05b0071970 0x7f05b015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05b606b640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b00af880 con 0x7f05b01579e0 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05b606b640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b0071e00 con 0x7f05b00b2ed0 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05b606b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b015a8c0 con 0x7f05b0071970 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05af7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f05b01579e0 0x7f05b00b4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50752/0 (socket says 172.21.15.28:50752) 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05af7fe640 1 -- 172.21.15.28:0/106437219 learned_addr learned my addr 172.21.15.28:0/106437219 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05aeffd640 1 --2- 172.21.15.28:0/106437219 >> v2:172.21.15.28:3300/0 conn(0x7f05b00b2ed0 0x7f05b0071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05affff640 1 --2- 172.21.15.28:0/106437219 >> v2:172.21.15.122:3300/0 conn(0x7f05b0071970 0x7f05b015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05ae7fc640 1 -- 172.21.15.28:0/106437219 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631826846 0 0) 0x7f05b00af880 con 0x7f05b01579e0 2023-12-16T21:57:47.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05ae7fc640 1 -- 172.21.15.28:0/106437219 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0590003530 con 0x7f05b01579e0 2023-12-16T21:57:47.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05aeffd640 1 -- 172.21.15.28:0/106437219 >> v2:172.21.15.122:3300/0 conn(0x7f05b0071970 msgr2=0x7f05b015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:47.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05aeffd640 1 --2- 172.21.15.28:0/106437219 >> v2:172.21.15.122:3300/0 conn(0x7f05b0071970 0x7f05b015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05aeffd640 1 -- 172.21.15.28:0/106437219 >> v1:172.21.15.28:6789/0 conn(0x7f05b01579e0 legacy=0x7f05b00b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.052+0000 7f05aeffd640 1 -- 172.21.15.28:0/106437219 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b015aba0 con 0x7f05b00b2ed0 2023-12-16T21:57:47.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05aeffd640 1 --2- 172.21.15.28:0/106437219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b00b2ed0 0x7f05b0071430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f05b00cd1c0 tx=0x7f059801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55576cd667f897a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05ae7fc640 1 -- 172.21.15.28:0/106437219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05980ac600 con 0x7f05b00b2ed0 2023-12-16T21:57:47.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05ae7fc640 1 -- 172.21.15.28:0/106437219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05980a8050 con 0x7f05b00b2ed0 2023-12-16T21:57:47.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05ae7fc640 1 -- 172.21.15.28:0/106437219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05980ac980 con 0x7f05b00b2ed0 2023-12-16T21:57:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- 172.21.15.28:0/106437219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b00b2ed0 msgr2=0x7f05b0071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 --2- 172.21.15.28:0/106437219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b00b2ed0 0x7f05b0071430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f05b00cd1c0 tx=0x7f059801c3a0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- 172.21.15.28:0/106437219 shutdown_connections 2023-12-16T21:57:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 --2- 172.21.15.28:0/106437219 >> v2:172.21.15.122:3300/0 conn(0x7f05b0071970 0x7f05b015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 --2- 172.21.15.28:0/106437219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b00b2ed0 0x7f05b0071430 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- 172.21.15.28:0/106437219 >> 172.21.15.28:0/106437219 conn(0x7f05b00b1bd0 msgr2=0x7f05b00b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- 172.21.15.28:0/106437219 shutdown_connections 2023-12-16T21:57:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- 172.21.15.28:0/106437219 wait complete. 2023-12-16T21:57:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 Processor -- start 2023-12-16T21:57:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- start start 2023-12-16T21:57:47.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f05b0071970 0x7f05b014f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05b00b2ed0 0x7f05b0160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b01579e0 0x7f05b01503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b015ebb0 con 0x7f05b01579e0 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b014f7c0 con 0x7f05b00b2ed0 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05b606b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b0160990 con 0x7f05b0071970 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b01579e0 0x7f05b01503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b01579e0 0x7f05b01503f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57540/0 (socket says 172.21.15.28:57540) 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05affff640 1 -- 172.21.15.28:0/1270659232 learned_addr learned my addr 172.21.15.28:0/1270659232 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05af7fe640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f05b0071970 0x7f05b014f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05aeffd640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05b00b2ed0 0x7f05b0160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05affff640 1 -- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f05b0071970 msgr2=0x7f05b014f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05affff640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f05b0071970 0x7f05b014f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05affff640 1 -- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05b00b2ed0 msgr2=0x7f05b0160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05affff640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05b00b2ed0 0x7f05b0160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05affff640 1 -- 172.21.15.28:0/1270659232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598009420 con 0x7f05b01579e0 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.056+0000 7f05aeffd640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05b00b2ed0 0x7f05b0160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:47.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05affff640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b01579e0 0x7f05b01503f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f05a8007e90 tx=0x7f05a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05a7fff640 1 -- 172.21.15.28:0/1270659232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8014420 con 0x7f05b01579e0 2023-12-16T21:57:47.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05a7fff640 1 -- 172.21.15.28:0/1270659232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a8014580 con 0x7f05b01579e0 2023-12-16T21:57:47.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b015ee00 con 0x7f05b01579e0 2023-12-16T21:57:47.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05a7fff640 1 -- 172.21.15.28:0/1270659232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8014930 con 0x7f05b01579e0 2023-12-16T21:57:47.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b015f3c0 con 0x7f05b01579e0 2023-12-16T21:57:47.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b00674d0 con 0x7f05b01579e0 2023-12-16T21:57:47.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05a7fff640 1 -- 172.21.15.28:0/1270659232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f05a8014a90 con 0x7f05b01579e0 2023-12-16T21:57:47.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05a7fff640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0594066470 0x7f0594068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05a7fff640 1 -- 172.21.15.28:0/1270659232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05a80847e0 con 0x7f05b01579e0 2023-12-16T21:57:47.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05af7fe640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0594066470 0x7f0594068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.060+0000 7f05af7fe640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0594066470 0x7f0594068930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f05a00011f0 tx=0x7f05a0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.064+0000 7f05a7fff640 1 -- 172.21.15.28:0/1270659232 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f05a8051c60 con 0x7f05b01579e0 2023-12-16T21:57:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.220+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05b0153530 con 0x7f05b01579e0 2023-12-16T21:57:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.220+0000 7f05a7fff640 1 -- 172.21.15.28:0/1270659232 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f05a8055280 con 0x7f05b01579e0 2023-12-16T21:57:47.221 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:47.221 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:47.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0594066470 msgr2=0x7f0594068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0594066470 0x7f0594068930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f05a00011f0 tx=0x7f05a0009040 comp rx=0 tx=0).stop 2023-12-16T21:57:47.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b01579e0 msgr2=0x7f05b01503f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b01579e0 0x7f05b01503f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f05a8007e90 tx=0x7f05a800e4b0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 shutdown_connections 2023-12-16T21:57:47.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0594066470 0x7f0594068930 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b01579e0 0x7f05b01503f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f05b00b2ed0 0x7f05b0160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 --2- 172.21.15.28:0/1270659232 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f05b0071970 0x7f05b014f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 >> 172.21.15.28:0/1270659232 conn(0x7f05b00b1bd0 msgr2=0x7f05b00b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:47.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 shutdown_connections 2023-12-16T21:57:47.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.224+0000 7f05b606b640 1 -- 172.21.15.28:0/1270659232 wait complete. 2023-12-16T21:57:47.239 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-16T21:57:47.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:57:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59d4b91640 1 Processor -- start 2023-12-16T21:57:47.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59d4b91640 1 -- start start 2023-12-16T21:57:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59d4b91640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0144840 0x7f59d006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59d4b91640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f59d0072180 0x7f59d006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59d4b91640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d00710e0 con 0x7f59d0070460 2023-12-16T21:57:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59d4b91640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d015f010 con 0x7f59d0072180 2023-12-16T21:57:47.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59d4b91640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00713c0 con 0x7f59d0144840 2023-12-16T21:57:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59c7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f59d0072180 0x7f59d006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59ce59c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0144840 0x7f59d006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59ced9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f59d0070460 0x7f59d00708e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:43652/0 (socket says 172.21.15.28:43652) 2023-12-16T21:57:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59ced9d640 1 -- 172.21.15.28:0/2373002788 learned_addr learned my addr 172.21.15.28:0/2373002788 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59ce59c640 1 -- 172.21.15.28:0/2373002788 >> v2:172.21.15.122:3300/0 conn(0x7f59d0072180 msgr2=0x7f59d006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59ce59c640 1 --2- 172.21.15.28:0/2373002788 >> v2:172.21.15.122:3300/0 conn(0x7f59d0072180 0x7f59d006c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59ce59c640 1 -- 172.21.15.28:0/2373002788 >> v1:172.21.15.122:6789/0 conn(0x7f59d0070460 legacy=0x7f59d00708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59ce59c640 1 -- 172.21.15.28:0/2373002788 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d00716a0 con 0x7f59d0144840 2023-12-16T21:57:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.360+0000 7f59c7fff640 1 --2- 172.21.15.28:0/2373002788 >> v2:172.21.15.122:3300/0 conn(0x7f59d0072180 0x7f59d006c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59ce59c640 1 --2- 172.21.15.28:0/2373002788 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0144840 0x7f59d006c2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f59c00097c0 tx=0x7f59c001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ef7729f532f847e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59cdd9b640 1 -- 172.21.15.28:0/2373002788 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00ab650 con 0x7f59d0144840 2023-12-16T21:57:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59cdd9b640 1 -- 172.21.15.28:0/2373002788 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c00ab7b0 con 0x7f59d0144840 2023-12-16T21:57:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59cdd9b640 1 -- 172.21.15.28:0/2373002788 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00abb40 con 0x7f59d0144840 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59d4b91640 1 -- 172.21.15.28:0/2373002788 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0144840 msgr2=0x7f59d006c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59d4b91640 1 --2- 172.21.15.28:0/2373002788 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0144840 0x7f59d006c2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f59c00097c0 tx=0x7f59c001c450 comp rx=0 tx=0).stop 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59d4b91640 1 -- 172.21.15.28:0/2373002788 shutdown_connections 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59d4b91640 1 --2- 172.21.15.28:0/2373002788 >> v2:172.21.15.122:3300/0 conn(0x7f59d0072180 0x7f59d006c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59d4b91640 1 --2- 172.21.15.28:0/2373002788 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0144840 0x7f59d006c2c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59d4b91640 1 -- 172.21.15.28:0/2373002788 >> 172.21.15.28:0/2373002788 conn(0x7f59d006a530 msgr2=0x7f59d006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59d4b91640 1 -- 172.21.15.28:0/2373002788 shutdown_connections 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.364+0000 7f59d4b91640 1 -- 172.21.15.28:0/2373002788 wait complete. 2023-12-16T21:57:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 Processor -- start 2023-12-16T21:57:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 -- start start 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0070460 0x7f59d015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0072180 0x7f59d0139e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59d0144840 0x7f59d013a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d0141e70 con 0x7f59d0072180 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d0136340 con 0x7f59d0144840 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00b3890 con 0x7f59d0070460 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0072180 0x7f59d0139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0072180 0x7f59d0139e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57546/0 (socket says 172.21.15.28:57546) 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 -- 172.21.15.28:0/4143862105 learned_addr learned my addr 172.21.15.28:0/4143862105 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59ced9d640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59d0144840 0x7f59d013a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59ce59c640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0070460 0x7f59d015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 -- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0070460 msgr2=0x7f59d015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0070460 0x7f59d015a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 -- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59d0144840 msgr2=0x7f59d013a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59d0144840 0x7f59d013a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 -- 172.21.15.28:0/4143862105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8009420 con 0x7f59d0072180 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c7fff640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0072180 0x7f59d0139e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59b800ebb0 tx=0x7f59b800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c6ffd640 1 -- 172.21.15.28:0/4143862105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b800eeb0 con 0x7f59d0072180 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c6ffd640 1 -- 172.21.15.28:0/4143862105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b800cc30 con 0x7f59d0072180 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c0009420 con 0x7f59d0072180 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c6ffd640 1 -- 172.21.15.28:0/4143862105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8013470 con 0x7f59d0072180 2023-12-16T21:57:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d01417b0 con 0x7f59d0072180 2023-12-16T21:57:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d01420e0 con 0x7f59d0072180 2023-12-16T21:57:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c6ffd640 1 -- 172.21.15.28:0/4143862105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f59b80135d0 con 0x7f59d0072180 2023-12-16T21:57:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.368+0000 7f59c6ffd640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f59b0066470 0x7f59b0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.372+0000 7f59c6ffd640 1 -- 172.21.15.28:0/4143862105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59b8084900 con 0x7f59d0072180 2023-12-16T21:57:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.372+0000 7f59ce59c640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f59b0066470 0x7f59b0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.372+0000 7f59ce59c640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f59b0066470 0x7f59b0068930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f59c00af000 tx=0x7f59c00a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.372+0000 7f59c6ffd640 1 -- 172.21.15.28:0/4143862105 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f59b8051e60 con 0x7f59d0072180 2023-12-16T21:57:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.528+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59d0069440 con 0x7f59b0066470 2023-12-16T21:57:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.552+0000 7f59c6ffd640 1 -- 172.21.15.28:0/4143862105 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264495 (secure 0 0 0) 0x7f59d0069440 con 0x7f59b0066470 2023-12-16T21:57:47.554 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:57:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f59b0066470 msgr2=0x7f59b0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f59b0066470 0x7f59b0068930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f59c00af000 tx=0x7f59c00a7000 comp rx=0 tx=0).stop 2023-12-16T21:57:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0072180 msgr2=0x7f59d0139e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0072180 0x7f59d0139e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59b800ebb0 tx=0x7f59b800c520 comp rx=0 tx=0).stop 2023-12-16T21:57:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 shutdown_connections 2023-12-16T21:57:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f59b0066470 0x7f59b0068930 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f59d0144840 0x7f59d013a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0072180 0x7f59d0139e10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 --2- 172.21.15.28:0/4143862105 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f59d0070460 0x7f59d015a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 >> 172.21.15.28:0/4143862105 conn(0x7f59d006a530 msgr2=0x7f59d015f240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 shutdown_connections 2023-12-16T21:57:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.556+0000 7f59d4b91640 1 -- 172.21.15.28:0/4143862105 wait complete. 2023-12-16T21:57:47.573 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-12-16T21:57:47.382660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235776,"kb_used_data":2980,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738944,"statfs":{"total":773094113280,"available":772852678656,"internally_reserved":0,"allocated":3051520,"data_stored":1805080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.918541"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264489+0000","last_change":"2023-12-16T21:57:43.518611+0000","last_active":"2023-12-16T21:57:45.264489+0000","last_peered":"2023-12-16T21:57:45.264489+0000","last_clean":"2023-12-16T21:57:45.264489+0000","last_became_active":"2023-12-16T21:57:43.517704+0000","last_became_peered":"2023-12-16T21:57:43.517704+0000","last_unstale":"2023-12-16T21:57:45.264489+0000","last_undegraded":"2023-12-16T21:57:45.264489+0000","last_fullsized":"2023-12-16T21:57:45.264489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:49:13.751852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870153+0000","last_change":"2023-12-16T21:57:43.485837+0000","last_active":"2023-12-16T21:57:44.870153+0000","last_peered":"2023-12-16T21:57:44.870153+0000","last_clean":"2023-12-16T21:57:44.870153+0000","last_became_active":"2023-12-16T21:57:43.484881+0000","last_became_peered":"2023-12-16T21:57:43.484881+0000","last_unstale":"2023-12-16T21:57:44.870153+0000","last_undegraded":"2023-12-16T21:57:44.870153+0000","last_fullsized":"2023-12-16T21:57:44.870153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252805+0000","last_change":"2023-12-16T21:57:43.492016+0000","last_active":"2023-12-16T21:57:45.252805+0000","last_peered":"2023-12-16T21:57:45.252805+0000","last_clean":"2023-12-16T21:57:45.252805+0000","last_became_active":"2023-12-16T21:57:43.491114+0000","last_became_peered":"2023-12-16T21:57:43.491114+0000","last_unstale":"2023-12-16T21:57:45.252805+0000","last_undegraded":"2023-12-16T21:57:45.252805+0000","last_fullsized":"2023-12-16T21:57:45.252805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262086+0000","last_change":"2023-12-16T21:57:43.510356+0000","last_active":"2023-12-16T21:57:45.262086+0000","last_peered":"2023-12-16T21:57:45.262086+0000","last_clean":"2023-12-16T21:57:45.262086+0000","last_became_active":"2023-12-16T21:57:43.509819+0000","last_became_peered":"2023-12-16T21:57:43.509819+0000","last_unstale":"2023-12-16T21:57:45.262086+0000","last_undegraded":"2023-12-16T21:57:45.262086+0000","last_fullsized":"2023-12-16T21:57:45.262086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264164+0000","last_change":"2023-12-16T21:57:43.507126+0000","last_active":"2023-12-16T21:57:45.264164+0000","last_peered":"2023-12-16T21:57:45.264164+0000","last_clean":"2023-12-16T21:57:45.264164+0000","last_became_active":"2023-12-16T21:57:43.506352+0000","last_became_peered":"2023-12-16T21:57:43.506352+0000","last_unstale":"2023-12-16T21:57:45.264164+0000","last_undegraded":"2023-12-16T21:57:45.264164+0000","last_fullsized":"2023-12-16T21:57:45.264164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:04:09.033093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262865+0000","last_change":"2023-12-16T21:57:43.483088+0000","last_active":"2023-12-16T21:57:45.262865+0000","last_peered":"2023-12-16T21:57:45.262865+0000","last_clean":"2023-12-16T21:57:45.262865+0000","last_became_active":"2023-12-16T21:57:43.482278+0000","last_became_peered":"2023-12-16T21:57:43.482278+0000","last_unstale":"2023-12-16T21:57:45.262865+0000","last_undegraded":"2023-12-16T21:57:45.262865+0000","last_fullsized":"2023-12-16T21:57:45.262865+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:15.360923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261922+0000","last_change":"2023-12-16T21:57:43.494598+0000","last_active":"2023-12-16T21:57:45.261922+0000","last_peered":"2023-12-16T21:57:45.261922+0000","last_clean":"2023-12-16T21:57:45.261922+0000","last_became_active":"2023-12-16T21:57:43.493867+0000","last_became_peered":"2023-12-16T21:57:43.493867+0000","last_unstale":"2023-12-16T21:57:45.261922+0000","last_undegraded":"2023-12-16T21:57:45.261922+0000","last_fullsized":"2023-12-16T21:57:45.261922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262085+0000","last_change":"2023-12-16T21:57:43.504864+0000","last_active":"2023-12-16T21:57:45.262085+0000","last_peered":"2023-12-16T21:57:45.262085+0000","last_clean":"2023-12-16T21:57:45.262085+0000","last_became_active":"2023-12-16T21:57:43.504140+0000","last_became_peered":"2023-12-16T21:57:43.504140+0000","last_unstale":"2023-12-16T21:57:45.262085+0000","last_undegraded":"2023-12-16T21:57:45.262085+0000","last_fullsized":"2023-12-16T21:57:45.262085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:22.190906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264734+0000","last_change":"2023-12-16T21:57:43.518225+0000","last_active":"2023-12-16T21:57:45.264734+0000","last_peered":"2023-12-16T21:57:45.264734+0000","last_clean":"2023-12-16T21:57:45.264734+0000","last_became_active":"2023-12-16T21:57:43.517602+0000","last_became_peered":"2023-12-16T21:57:43.517602+0000","last_unstale":"2023-12-16T21:57:45.264734+0000","last_undegraded":"2023-12-16T21:57:45.264734+0000","last_fullsized":"2023-12-16T21:57:45.264734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:46:04.514997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.460177+0000","last_change":"2023-12-16T21:57:43.468564+0000","last_active":"2023-12-16T21:57:44.460177+0000","last_peered":"2023-12-16T21:57:44.460177+0000","last_clean":"2023-12-16T21:57:44.460177+0000","last_became_active":"2023-12-16T21:57:43.467674+0000","last_became_peered":"2023-12-16T21:57:43.467674+0000","last_unstale":"2023-12-16T21:57:44.460177+0000","last_undegraded":"2023-12-16T21:57:44.460177+0000","last_fullsized":"2023-12-16T21:57:44.460177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458605+0000","last_change":"2023-12-16T21:57:43.474153+0000","last_active":"2023-12-16T21:57:44.458605+0000","last_peered":"2023-12-16T21:57:44.458605+0000","last_clean":"2023-12-16T21:57:44.458605+0000","last_became_active":"2023-12-16T21:57:43.467788+0000","last_became_peered":"2023-12-16T21:57:43.467788+0000","last_unstale":"2023-12-16T21:57:44.458605+0000","last_undegraded":"2023-12-16T21:57:44.458605+0000","last_fullsized":"2023-12-16T21:57:44.458605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458900+0000","last_change":"2023-12-16T21:57:43.494364+0000","last_active":"2023-12-16T21:57:44.458900+0000","last_peered":"2023-12-16T21:57:44.458900+0000","last_clean":"2023-12-16T21:57:44.458900+0000","last_became_active":"2023-12-16T21:57:43.491952+0000","last_became_peered":"2023-12-16T21:57:43.491952+0000","last_unstale":"2023-12-16T21:57:44.458900+0000","last_undegraded":"2023-12-16T21:57:44.458900+0000","last_fullsized":"2023-12-16T21:57:44.458900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:16.932172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262657+0000","last_change":"2023-12-16T21:57:43.490348+0000","last_active":"2023-12-16T21:57:45.262657+0000","last_peered":"2023-12-16T21:57:45.262657+0000","last_clean":"2023-12-16T21:57:45.262657+0000","last_became_active":"2023-12-16T21:57:43.489582+0000","last_became_peered":"2023-12-16T21:57:43.489582+0000","last_unstale":"2023-12-16T21:57:45.262657+0000","last_undegraded":"2023-12-16T21:57:45.262657+0000","last_fullsized":"2023-12-16T21:57:45.262657+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252238+0000","last_change":"2023-12-16T21:57:43.481446+0000","last_active":"2023-12-16T21:57:45.252238+0000","last_peered":"2023-12-16T21:57:45.252238+0000","last_clean":"2023-12-16T21:57:45.252238+0000","last_became_active":"2023-12-16T21:57:43.480422+0000","last_became_peered":"2023-12-16T21:57:43.480422+0000","last_unstale":"2023-12-16T21:57:45.252238+0000","last_undegraded":"2023-12-16T21:57:45.252238+0000","last_fullsized":"2023-12-16T21:57:45.252238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869559+0000","last_change":"2023-12-16T21:57:43.507792+0000","last_active":"2023-12-16T21:57:44.869559+0000","last_peered":"2023-12-16T21:57:44.869559+0000","last_clean":"2023-12-16T21:57:44.869559+0000","last_became_active":"2023-12-16T21:57:43.507030+0000","last_became_peered":"2023-12-16T21:57:43.507030+0000","last_unstale":"2023-12-16T21:57:44.869559+0000","last_undegraded":"2023-12-16T21:57:44.869559+0000","last_fullsized":"2023-12-16T21:57:44.869559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261659+0000","last_change":"2023-12-16T21:57:43.485531+0000","last_active":"2023-12-16T21:57:45.261659+0000","last_peered":"2023-12-16T21:57:45.261659+0000","last_clean":"2023-12-16T21:57:45.261659+0000","last_became_active":"2023-12-16T21:57:43.482538+0000","last_became_peered":"2023-12-16T21:57:43.482538+0000","last_unstale":"2023-12-16T21:57:45.261659+0000","last_undegraded":"2023-12-16T21:57:45.261659+0000","last_fullsized":"2023-12-16T21:57:45.261659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263056+0000","last_change":"2023-12-16T21:57:43.507572+0000","last_active":"2023-12-16T21:57:45.263056+0000","last_peered":"2023-12-16T21:57:45.263056+0000","last_clean":"2023-12-16T21:57:45.263056+0000","last_became_active":"2023-12-16T21:57:43.507072+0000","last_became_peered":"2023-12-16T21:57:43.507072+0000","last_unstale":"2023-12-16T21:57:45.263056+0000","last_undegraded":"2023-12-16T21:57:45.263056+0000","last_fullsized":"2023-12-16T21:57:45.263056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:49:13.751852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.502007+0000","last_change":"2023-12-16T21:57:43.487657+0000","last_active":"2023-12-16T21:57:43.502007+0000","last_peered":"2023-12-16T21:57:43.502007+0000","last_clean":"2023-12-16T21:57:43.502007+0000","last_became_active":"2023-12-16T21:57:43.484848+0000","last_became_peered":"2023-12-16T21:57:43.484848+0000","last_unstale":"2023-12-16T21:57:43.502007+0000","last_undegraded":"2023-12-16T21:57:43.502007+0000","last_fullsized":"2023-12-16T21:57:43.502007+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:30:24.150158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264990+0000","last_change":"2023-12-16T21:57:43.503136+0000","last_active":"2023-12-16T21:57:45.264990+0000","last_peered":"2023-12-16T21:57:45.264990+0000","last_clean":"2023-12-16T21:57:45.264990+0000","last_became_active":"2023-12-16T21:57:43.502321+0000","last_became_peered":"2023-12-16T21:57:43.502321+0000","last_unstale":"2023-12-16T21:57:45.264990+0000","last_undegraded":"2023-12-16T21:57:45.264990+0000","last_fullsized":"2023-12-16T21:57:45.264990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:29.457727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458117+0000","last_change":"2023-12-16T21:57:43.496151+0000","last_active":"2023-12-16T21:57:44.458117+0000","last_peered":"2023-12-16T21:57:44.458117+0000","last_clean":"2023-12-16T21:57:44.458117+0000","last_became_active":"2023-12-16T21:57:43.495293+0000","last_became_peered":"2023-12-16T21:57:43.495293+0000","last_unstale":"2023-12-16T21:57:44.458117+0000","last_undegraded":"2023-12-16T21:57:44.458117+0000","last_fullsized":"2023-12-16T21:57:44.458117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252271+0000","last_change":"2023-12-16T21:57:43.507839+0000","last_active":"2023-12-16T21:57:45.252271+0000","last_peered":"2023-12-16T21:57:45.252271+0000","last_clean":"2023-12-16T21:57:45.252271+0000","last_became_active":"2023-12-16T21:57:43.507150+0000","last_became_peered":"2023-12-16T21:57:43.507150+0000","last_unstale":"2023-12-16T21:57:45.252271+0000","last_undegraded":"2023-12-16T21:57:45.252271+0000","last_fullsized":"2023-12-16T21:57:45.252271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263163+0000","last_change":"2023-12-16T21:57:43.487735+0000","last_active":"2023-12-16T21:57:45.263163+0000","last_peered":"2023-12-16T21:57:45.263163+0000","last_clean":"2023-12-16T21:57:45.263163+0000","last_became_active":"2023-12-16T21:57:43.486972+0000","last_became_peered":"2023-12-16T21:57:43.486972+0000","last_unstale":"2023-12-16T21:57:45.263163+0000","last_undegraded":"2023-12-16T21:57:45.263163+0000","last_fullsized":"2023-12-16T21:57:45.263163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:46:04.514997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262348+0000","last_change":"2023-12-16T21:57:43.513648+0000","last_active":"2023-12-16T21:57:45.262348+0000","last_peered":"2023-12-16T21:57:45.262348+0000","last_clean":"2023-12-16T21:57:45.262348+0000","last_became_active":"2023-12-16T21:57:43.513086+0000","last_became_peered":"2023-12-16T21:57:43.513086+0000","last_unstale":"2023-12-16T21:57:45.262348+0000","last_undegraded":"2023-12-16T21:57:45.262348+0000","last_fullsized":"2023-12-16T21:57:45.262348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261798+0000","last_change":"2023-12-16T21:57:43.505290+0000","last_active":"2023-12-16T21:57:45.261798+0000","last_peered":"2023-12-16T21:57:45.261798+0000","last_clean":"2023-12-16T21:57:45.261798+0000","last_became_active":"2023-12-16T21:57:43.504600+0000","last_became_peered":"2023-12-16T21:57:43.504600+0000","last_unstale":"2023-12-16T21:57:45.261798+0000","last_undegraded":"2023-12-16T21:57:45.261798+0000","last_fullsized":"2023-12-16T21:57:45.261798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263041+0000","last_change":"2023-12-16T21:57:43.510116+0000","last_active":"2023-12-16T21:57:45.263041+0000","last_peered":"2023-12-16T21:57:45.263041+0000","last_clean":"2023-12-16T21:57:45.263041+0000","last_became_active":"2023-12-16T21:57:43.509260+0000","last_became_peered":"2023-12-16T21:57:43.509260+0000","last_unstale":"2023-12-16T21:57:45.263041+0000","last_undegraded":"2023-12-16T21:57:45.263041+0000","last_fullsized":"2023-12-16T21:57:45.263041+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.460344+0000","last_change":"2023-12-16T21:57:43.489237+0000","last_active":"2023-12-16T21:57:44.460344+0000","last_peered":"2023-12-16T21:57:44.460344+0000","last_clean":"2023-12-16T21:57:44.460344+0000","last_became_active":"2023-12-16T21:57:43.488252+0000","last_became_peered":"2023-12-16T21:57:43.488252+0000","last_unstale":"2023-12-16T21:57:44.460344+0000","last_undegraded":"2023-12-16T21:57:44.460344+0000","last_fullsized":"2023-12-16T21:57:44.460344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869771+0000","last_change":"2023-12-16T21:57:43.502339+0000","last_active":"2023-12-16T21:57:44.869771+0000","last_peered":"2023-12-16T21:57:44.869771+0000","last_clean":"2023-12-16T21:57:44.869771+0000","last_became_active":"2023-12-16T21:57:43.501434+0000","last_became_peered":"2023-12-16T21:57:43.501434+0000","last_unstale":"2023-12-16T21:57:44.869771+0000","last_undegraded":"2023-12-16T21:57:44.869771+0000","last_fullsized":"2023-12-16T21:57:44.869771+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263170+0000","last_change":"2023-12-16T21:57:43.509617+0000","last_active":"2023-12-16T21:57:45.263170+0000","last_peered":"2023-12-16T21:57:45.263170+0000","last_clean":"2023-12-16T21:57:45.263170+0000","last_became_active":"2023-12-16T21:57:43.508922+0000","last_became_peered":"2023-12-16T21:57:43.508922+0000","last_unstale":"2023-12-16T21:57:45.263170+0000","last_undegraded":"2023-12-16T21:57:45.263170+0000","last_fullsized":"2023-12-16T21:57:45.263170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252408+0000","last_change":"2023-12-16T21:57:43.505933+0000","last_active":"2023-12-16T21:57:45.252408+0000","last_peered":"2023-12-16T21:57:45.252408+0000","last_clean":"2023-12-16T21:57:45.252408+0000","last_became_active":"2023-12-16T21:57:43.504463+0000","last_became_peered":"2023-12-16T21:57:43.504463+0000","last_unstale":"2023-12-16T21:57:45.252408+0000","last_undegraded":"2023-12-16T21:57:45.252408+0000","last_fullsized":"2023-12-16T21:57:45.252408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263332+0000","last_change":"2023-12-16T21:57:43.501738+0000","last_active":"2023-12-16T21:57:45.263332+0000","last_peered":"2023-12-16T21:57:45.263332+0000","last_clean":"2023-12-16T21:57:45.263332+0000","last_became_active":"2023-12-16T21:57:43.501014+0000","last_became_peered":"2023-12-16T21:57:43.501014+0000","last_unstale":"2023-12-16T21:57:45.263332+0000","last_undegraded":"2023-12-16T21:57:45.263332+0000","last_fullsized":"2023-12-16T21:57:45.263332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:47:38.932092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.480394+0000","last_change":"2023-12-16T21:57:43.477348+0000","last_active":"2023-12-16T21:57:43.480394+0000","last_peered":"2023-12-16T21:57:43.480394+0000","last_clean":"2023-12-16T21:57:43.480394+0000","last_became_active":"2023-12-16T21:57:43.476616+0000","last_became_peered":"2023-12-16T21:57:43.476616+0000","last_unstale":"2023-12-16T21:57:43.480394+0000","last_undegraded":"2023-12-16T21:57:43.480394+0000","last_fullsized":"2023-12-16T21:57:43.480394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:48:17.536347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457542+0000","last_change":"2023-12-16T21:57:43.483725+0000","last_active":"2023-12-16T21:57:44.457542+0000","last_peered":"2023-12-16T21:57:44.457542+0000","last_clean":"2023-12-16T21:57:44.457542+0000","last_became_active":"2023-12-16T21:57:43.482870+0000","last_became_peered":"2023-12-16T21:57:43.482870+0000","last_unstale":"2023-12-16T21:57:44.457542+0000","last_undegraded":"2023-12-16T21:57:44.457542+0000","last_fullsized":"2023-12-16T21:57:44.457542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458048+0000","last_change":"2023-12-16T21:57:43.492281+0000","last_active":"2023-12-16T21:57:44.458048+0000","last_peered":"2023-12-16T21:57:44.458048+0000","last_clean":"2023-12-16T21:57:44.458048+0000","last_became_active":"2023-12-16T21:57:43.491538+0000","last_became_peered":"2023-12-16T21:57:43.491538+0000","last_unstale":"2023-12-16T21:57:44.458048+0000","last_undegraded":"2023-12-16T21:57:44.458048+0000","last_fullsized":"2023-12-16T21:57:44.458048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:21:57.846367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262966+0000","last_change":"2023-12-16T21:57:42.522074+0000","last_active":"2023-12-16T21:57:45.262966+0000","last_peered":"2023-12-16T21:57:45.262966+0000","last_clean":"2023-12-16T21:57:45.262966+0000","last_became_active":"2023-12-16T21:57:42.521769+0000","last_became_peered":"2023-12-16T21:57:42.521769+0000","last_unstale":"2023-12-16T21:57:45.262966+0000","last_undegraded":"2023-12-16T21:57:45.262966+0000","last_fullsized":"2023-12-16T21:57:45.262966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:42:59.000744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252714+0000","last_change":"2023-12-16T21:57:43.495840+0000","last_active":"2023-12-16T21:57:45.252714+0000","last_peered":"2023-12-16T21:57:45.252714+0000","last_clean":"2023-12-16T21:57:45.252714+0000","last_became_active":"2023-12-16T21:57:43.495114+0000","last_became_peered":"2023-12-16T21:57:43.495114+0000","last_unstale":"2023-12-16T21:57:45.252714+0000","last_undegraded":"2023-12-16T21:57:45.252714+0000","last_fullsized":"2023-12-16T21:57:45.252714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458001+0000","last_change":"2023-12-16T21:57:42.515225+0000","last_active":"2023-12-16T21:57:44.458001+0000","last_peered":"2023-12-16T21:57:44.458001+0000","last_clean":"2023-12-16T21:57:44.458001+0000","last_became_active":"2023-12-16T21:57:42.514509+0000","last_became_peered":"2023-12-16T21:57:42.514509+0000","last_unstale":"2023-12-16T21:57:44.458001+0000","last_undegraded":"2023-12-16T21:57:44.458001+0000","last_fullsized":"2023-12-16T21:57:44.458001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:21.179733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262272+0000","last_change":"2023-12-16T21:57:42.515658+0000","last_active":"2023-12-16T21:57:45.262272+0000","last_peered":"2023-12-16T21:57:45.262272+0000","last_clean":"2023-12-16T21:57:45.262272+0000","last_became_active":"2023-12-16T21:57:42.515275+0000","last_became_peered":"2023-12-16T21:57:42.515275+0000","last_unstale":"2023-12-16T21:57:45.262272+0000","last_undegraded":"2023-12-16T21:57:45.262272+0000","last_fullsized":"2023-12-16T21:57:45.262272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:00:29.774694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262293+0000","last_change":"2023-12-16T21:57:43.493987+0000","last_active":"2023-12-16T21:57:45.262293+0000","last_peered":"2023-12-16T21:57:45.262293+0000","last_clean":"2023-12-16T21:57:45.262293+0000","last_became_active":"2023-12-16T21:57:43.493193+0000","last_became_peered":"2023-12-16T21:57:43.493193+0000","last_unstale":"2023-12-16T21:57:45.262293+0000","last_undegraded":"2023-12-16T21:57:45.262293+0000","last_fullsized":"2023-12-16T21:57:45.262293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261893+0000","last_change":"2023-12-16T21:57:42.508909+0000","last_active":"2023-12-16T21:57:45.261893+0000","last_peered":"2023-12-16T21:57:45.261893+0000","last_clean":"2023-12-16T21:57:45.261893+0000","last_became_active":"2023-12-16T21:57:42.508250+0000","last_became_peered":"2023-12-16T21:57:42.508250+0000","last_unstale":"2023-12-16T21:57:45.261893+0000","last_undegraded":"2023-12-16T21:57:45.261893+0000","last_fullsized":"2023-12-16T21:57:45.261893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:39.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869537+0000","last_change":"2023-12-16T21:57:43.487695+0000","last_active":"2023-12-16T21:57:44.869537+0000","last_peered":"2023-12-16T21:57:44.869537+0000","last_clean":"2023-12-16T21:57:44.869537+0000","last_became_active":"2023-12-16T21:57:43.486936+0000","last_became_peered":"2023-12-16T21:57:43.486936+0000","last_unstale":"2023-12-16T21:57:44.869537+0000","last_undegraded":"2023-12-16T21:57:44.869537+0000","last_fullsized":"2023-12-16T21:57:44.869537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458471+0000","last_change":"2023-12-16T21:57:42.526210+0000","last_active":"2023-12-16T21:57:44.458471+0000","last_peered":"2023-12-16T21:57:44.458471+0000","last_clean":"2023-12-16T21:57:44.458471+0000","last_became_active":"2023-12-16T21:57:42.525460+0000","last_became_peered":"2023-12-16T21:57:42.525460+0000","last_unstale":"2023-12-16T21:57:44.458471+0000","last_undegraded":"2023-12-16T21:57:44.458471+0000","last_fullsized":"2023-12-16T21:57:44.458471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:43:36.373508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263137+0000","last_change":"2023-12-16T21:57:43.483521+0000","last_active":"2023-12-16T21:57:45.263137+0000","last_peered":"2023-12-16T21:57:45.263137+0000","last_clean":"2023-12-16T21:57:45.263137+0000","last_became_active":"2023-12-16T21:57:43.482802+0000","last_became_peered":"2023-12-16T21:57:43.482802+0000","last_unstale":"2023-12-16T21:57:45.263137+0000","last_undegraded":"2023-12-16T21:57:45.263137+0000","last_fullsized":"2023-12-16T21:57:45.263137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:29.175289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263361+0000","last_change":"2023-12-16T21:57:42.517029+0000","last_active":"2023-12-16T21:57:45.263361+0000","last_peered":"2023-12-16T21:57:45.263361+0000","last_clean":"2023-12-16T21:57:45.263361+0000","last_became_active":"2023-12-16T21:57:42.516592+0000","last_became_peered":"2023-12-16T21:57:42.516592+0000","last_unstale":"2023-12-16T21:57:45.263361+0000","last_undegraded":"2023-12-16T21:57:45.263361+0000","last_fullsized":"2023-12-16T21:57:45.263361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:42:59.000744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252990+0000","last_change":"2023-12-16T21:57:43.480932+0000","last_active":"2023-12-16T21:57:45.252990+0000","last_peered":"2023-12-16T21:57:45.252990+0000","last_clean":"2023-12-16T21:57:45.252990+0000","last_became_active":"2023-12-16T21:57:43.480056+0000","last_became_peered":"2023-12-16T21:57:43.480056+0000","last_unstale":"2023-12-16T21:57:45.252990+0000","last_undegraded":"2023-12-16T21:57:45.252990+0000","last_fullsized":"2023-12-16T21:57:45.252990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869739+0000","last_change":"2023-12-16T21:57:42.499117+0000","last_active":"2023-12-16T21:57:44.869739+0000","last_peered":"2023-12-16T21:57:44.869739+0000","last_clean":"2023-12-16T21:57:44.869739+0000","last_became_active":"2023-12-16T21:57:42.498315+0000","last_became_peered":"2023-12-16T21:57:42.498315+0000","last_unstale":"2023-12-16T21:57:44.869739+0000","last_undegraded":"2023-12-16T21:57:44.869739+0000","last_fullsized":"2023-12-16T21:57:44.869739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:14:36.842626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459102+0000","last_change":"2023-12-16T21:57:43.507811+0000","last_active":"2023-12-16T21:57:44.459102+0000","last_peered":"2023-12-16T21:57:44.459102+0000","last_clean":"2023-12-16T21:57:44.459102+0000","last_became_active":"2023-12-16T21:57:43.507089+0000","last_became_peered":"2023-12-16T21:57:43.507089+0000","last_unstale":"2023-12-16T21:57:44.459102+0000","last_undegraded":"2023-12-16T21:57:44.459102+0000","last_fullsized":"2023-12-16T21:57:44.459102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:43:00.011918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458372+0000","last_change":"2023-12-16T21:57:42.512797+0000","last_active":"2023-12-16T21:57:44.458372+0000","last_peered":"2023-12-16T21:57:44.458372+0000","last_clean":"2023-12-16T21:57:44.458372+0000","last_became_active":"2023-12-16T21:57:42.512055+0000","last_became_peered":"2023-12-16T21:57:42.512055+0000","last_unstale":"2023-12-16T21:57:44.458372+0000","last_undegraded":"2023-12-16T21:57:44.458372+0000","last_fullsized":"2023-12-16T21:57:44.458372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:15.920999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264337+0000","last_change":"2023-12-16T21:57:43.510475+0000","last_active":"2023-12-16T21:57:45.264337+0000","last_peered":"2023-12-16T21:57:45.264337+0000","last_clean":"2023-12-16T21:57:45.264337+0000","last_became_active":"2023-12-16T21:57:43.509720+0000","last_became_peered":"2023-12-16T21:57:43.509720+0000","last_unstale":"2023-12-16T21:57:45.264337+0000","last_undegraded":"2023-12-16T21:57:45.264337+0000","last_fullsized":"2023-12-16T21:57:45.264337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:15.360923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252634+0000","last_change":"2023-12-16T21:57:42.518031+0000","last_active":"2023-12-16T21:57:45.252634+0000","last_peered":"2023-12-16T21:57:45.252634+0000","last_clean":"2023-12-16T21:57:45.252634+0000","last_became_active":"2023-12-16T21:57:42.517664+0000","last_became_peered":"2023-12-16T21:57:42.517664+0000","last_unstale":"2023-12-16T21:57:45.252634+0000","last_undegraded":"2023-12-16T21:57:45.252634+0000","last_fullsized":"2023-12-16T21:57:45.252634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252215+0000","last_change":"2023-12-16T21:57:43.493896+0000","last_active":"2023-12-16T21:57:45.252215+0000","last_peered":"2023-12-16T21:57:45.252215+0000","last_clean":"2023-12-16T21:57:45.252215+0000","last_became_active":"2023-12-16T21:57:43.492917+0000","last_became_peered":"2023-12-16T21:57:43.492917+0000","last_unstale":"2023-12-16T21:57:45.252215+0000","last_undegraded":"2023-12-16T21:57:45.252215+0000","last_fullsized":"2023-12-16T21:57:45.252215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:22.190906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263439+0000","last_change":"2023-12-16T21:57:42.517161+0000","last_active":"2023-12-16T21:57:45.263439+0000","last_peered":"2023-12-16T21:57:45.263439+0000","last_clean":"2023-12-16T21:57:45.263439+0000","last_became_active":"2023-12-16T21:57:42.516388+0000","last_became_peered":"2023-12-16T21:57:42.516388+0000","last_unstale":"2023-12-16T21:57:45.263439+0000","last_undegraded":"2023-12-16T21:57:45.263439+0000","last_fullsized":"2023-12-16T21:57:45.263439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:15.423681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264304+0000","last_change":"2023-12-16T21:57:43.512923+0000","last_active":"2023-12-16T21:57:45.264304+0000","last_peered":"2023-12-16T21:57:45.264304+0000","last_clean":"2023-12-16T21:57:45.264304+0000","last_became_active":"2023-12-16T21:57:43.512254+0000","last_became_peered":"2023-12-16T21:57:43.512254+0000","last_unstale":"2023-12-16T21:57:45.264304+0000","last_undegraded":"2023-12-16T21:57:45.264304+0000","last_fullsized":"2023-12-16T21:57:45.264304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:20.834561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457916+0000","last_change":"2023-12-16T21:57:42.511480+0000","last_active":"2023-12-16T21:57:44.457916+0000","last_peered":"2023-12-16T21:57:44.457916+0000","last_clean":"2023-12-16T21:57:44.457916+0000","last_became_active":"2023-12-16T21:57:42.510772+0000","last_became_peered":"2023-12-16T21:57:42.510772+0000","last_unstale":"2023-12-16T21:57:44.457916+0000","last_undegraded":"2023-12-16T21:57:44.457916+0000","last_fullsized":"2023-12-16T21:57:44.457916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457878+0000","last_change":"2023-12-16T21:57:43.493903+0000","last_active":"2023-12-16T21:57:44.457878+0000","last_peered":"2023-12-16T21:57:44.457878+0000","last_clean":"2023-12-16T21:57:44.457878+0000","last_became_active":"2023-12-16T21:57:43.493169+0000","last_became_peered":"2023-12-16T21:57:43.493169+0000","last_unstale":"2023-12-16T21:57:44.457878+0000","last_undegraded":"2023-12-16T21:57:44.457878+0000","last_fullsized":"2023-12-16T21:57:44.457878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.538705+0000","last_change":"2023-12-16T21:57:42.498884+0000","last_active":"2023-12-16T21:57:43.538705+0000","last_peered":"2023-12-16T21:57:43.538705+0000","last_clean":"2023-12-16T21:57:43.538705+0000","last_became_active":"2023-12-16T21:57:42.498452+0000","last_became_peered":"2023-12-16T21:57:42.498452+0000","last_unstale":"2023-12-16T21:57:43.538705+0000","last_undegraded":"2023-12-16T21:57:43.538705+0000","last_fullsized":"2023-12-16T21:57:43.538705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:37:03.298907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263857+0000","last_change":"2023-12-16T21:57:43.513792+0000","last_active":"2023-12-16T21:57:45.263857+0000","last_peered":"2023-12-16T21:57:45.263857+0000","last_clean":"2023-12-16T21:57:45.263857+0000","last_became_active":"2023-12-16T21:57:43.513001+0000","last_became_peered":"2023-12-16T21:57:43.513001+0000","last_unstale":"2023-12-16T21:57:45.263857+0000","last_undegraded":"2023-12-16T21:57:45.263857+0000","last_fullsized":"2023-12-16T21:57:45.263857+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459156+0000","last_change":"2023-12-16T21:57:42.512676+0000","last_active":"2023-12-16T21:57:44.459156+0000","last_peered":"2023-12-16T21:57:44.459156+0000","last_clean":"2023-12-16T21:57:44.459156+0000","last_became_active":"2023-12-16T21:57:42.512267+0000","last_became_peered":"2023-12-16T21:57:42.512267+0000","last_unstale":"2023-12-16T21:57:44.459156+0000","last_undegraded":"2023-12-16T21:57:44.459156+0000","last_fullsized":"2023-12-16T21:57:44.459156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:23:06.394772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264897+0000","last_change":"2023-12-16T21:57:43.517003+0000","last_active":"2023-12-16T21:57:45.264897+0000","last_peered":"2023-12-16T21:57:45.264897+0000","last_clean":"2023-12-16T21:57:45.264897+0000","last_became_active":"2023-12-16T21:57:43.516197+0000","last_became_peered":"2023-12-16T21:57:43.516197+0000","last_unstale":"2023-12-16T21:57:45.264897+0000","last_undegraded":"2023-12-16T21:57:45.264897+0000","last_fullsized":"2023-12-16T21:57:45.264897+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:48:39.071922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.534245+0000","last_change":"2023-12-16T21:57:42.512911+0000","last_active":"2023-12-16T21:57:43.534245+0000","last_peered":"2023-12-16T21:57:43.534245+0000","last_clean":"2023-12-16T21:57:43.534245+0000","last_became_active":"2023-12-16T21:57:42.512410+0000","last_became_peered":"2023-12-16T21:57:42.512410+0000","last_unstale":"2023-12-16T21:57:43.534245+0000","last_undegraded":"2023-12-16T21:57:43.534245+0000","last_fullsized":"2023-12-16T21:57:43.534245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:00:58.952738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264838+0000","last_change":"2023-12-16T21:57:43.518693+0000","last_active":"2023-12-16T21:57:45.264838+0000","last_peered":"2023-12-16T21:57:45.264838+0000","last_clean":"2023-12-16T21:57:45.264838+0000","last_became_active":"2023-12-16T21:57:43.518184+0000","last_became_peered":"2023-12-16T21:57:43.518184+0000","last_unstale":"2023-12-16T21:57:45.264838+0000","last_undegraded":"2023-12-16T21:57:45.264838+0000","last_fullsized":"2023-12-16T21:57:45.264838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:47:38.932092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869962+0000","last_change":"2023-12-16T21:57:42.505084+0000","last_active":"2023-12-16T21:57:44.869962+0000","last_peered":"2023-12-16T21:57:44.869962+0000","last_clean":"2023-12-16T21:57:44.869962+0000","last_became_active":"2023-12-16T21:57:42.502948+0000","last_became_peered":"2023-12-16T21:57:42.502948+0000","last_unstale":"2023-12-16T21:57:44.869962+0000","last_undegraded":"2023-12-16T21:57:44.869962+0000","last_fullsized":"2023-12-16T21:57:44.869962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262484+0000","last_change":"2023-12-16T21:57:43.499808+0000","last_active":"2023-12-16T21:57:45.262484+0000","last_peered":"2023-12-16T21:57:45.262484+0000","last_clean":"2023-12-16T21:57:45.262484+0000","last_became_active":"2023-12-16T21:57:43.498922+0000","last_became_peered":"2023-12-16T21:57:43.498922+0000","last_unstale":"2023-12-16T21:57:45.262484+0000","last_undegraded":"2023-12-16T21:57:45.262484+0000","last_fullsized":"2023-12-16T21:57:45.262484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:04:09.033093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.465095+0000","last_change":"2023-12-16T21:57:42.509838+0000","last_active":"2023-12-16T21:57:43.465095+0000","last_peered":"2023-12-16T21:57:43.465095+0000","last_clean":"2023-12-16T21:57:43.465095+0000","last_became_active":"2023-12-16T21:57:42.508994+0000","last_became_peered":"2023-12-16T21:57:42.508994+0000","last_unstale":"2023-12-16T21:57:43.465095+0000","last_undegraded":"2023-12-16T21:57:43.465095+0000","last_fullsized":"2023-12-16T21:57:43.465095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:03:03.258824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264723+0000","last_change":"2023-12-16T21:57:43.516388+0000","last_active":"2023-12-16T21:57:45.264723+0000","last_peered":"2023-12-16T21:57:45.264723+0000","last_clean":"2023-12-16T21:57:45.264723+0000","last_became_active":"2023-12-16T21:57:43.515749+0000","last_became_peered":"2023-12-16T21:57:43.515749+0000","last_unstale":"2023-12-16T21:57:45.264723+0000","last_undegraded":"2023-12-16T21:57:45.264723+0000","last_fullsized":"2023-12-16T21:57:45.264723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:29.175289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.474023+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:57:43.474023+0000","last_peered":"2023-12-16T21:57:43.474023+0000","last_clean":"2023-12-16T21:57:43.474023+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:57:43.474023+0000","last_undegraded":"2023-12-16T21:57:43.474023+0000","last_fullsized":"2023-12-16T21:57:43.474023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261664+0000","last_change":"2023-12-16T21:57:42.508124+0000","last_active":"2023-12-16T21:57:45.261664+0000","last_peered":"2023-12-16T21:57:45.261664+0000","last_clean":"2023-12-16T21:57:45.261664+0000","last_became_active":"2023-12-16T21:57:42.507727+0000","last_became_peered":"2023-12-16T21:57:42.507727+0000","last_unstale":"2023-12-16T21:57:45.261664+0000","last_undegraded":"2023-12-16T21:57:45.261664+0000","last_fullsized":"2023-12-16T21:57:45.261664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262915+0000","last_change":"2023-12-16T21:57:43.490787+0000","last_active":"2023-12-16T21:57:45.262915+0000","last_peered":"2023-12-16T21:57:45.262915+0000","last_clean":"2023-12-16T21:57:45.262915+0000","last_became_active":"2023-12-16T21:57:43.490013+0000","last_became_peered":"2023-12-16T21:57:43.490013+0000","last_unstale":"2023-12-16T21:57:45.262915+0000","last_undegraded":"2023-12-16T21:57:45.262915+0000","last_fullsized":"2023-12-16T21:57:45.262915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262013+0000","last_change":"2023-12-16T21:57:42.523476+0000","last_active":"2023-12-16T21:57:45.262013+0000","last_peered":"2023-12-16T21:57:45.262013+0000","last_clean":"2023-12-16T21:57:45.262013+0000","last_became_active":"2023-12-16T21:57:42.522441+0000","last_became_peered":"2023-12-16T21:57:42.522441+0000","last_unstale":"2023-12-16T21:57:45.262013+0000","last_undegraded":"2023-12-16T21:57:45.262013+0000","last_fullsized":"2023-12-16T21:57:45.262013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:15.920999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252089+0000","last_change":"2023-12-16T21:57:43.508913+0000","last_active":"2023-12-16T21:57:45.252089+0000","last_peered":"2023-12-16T21:57:45.252089+0000","last_clean":"2023-12-16T21:57:45.252089+0000","last_became_active":"2023-12-16T21:57:43.508133+0000","last_became_peered":"2023-12-16T21:57:43.508133+0000","last_unstale":"2023-12-16T21:57:45.252089+0000","last_undegraded":"2023-12-16T21:57:45.252089+0000","last_fullsized":"2023-12-16T21:57:45.252089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:58.171558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.462961+0000","last_change":"2023-12-16T21:57:42.513117+0000","last_active":"2023-12-16T21:57:43.462961+0000","last_peered":"2023-12-16T21:57:43.462961+0000","last_clean":"2023-12-16T21:57:43.462961+0000","last_became_active":"2023-12-16T21:57:42.512449+0000","last_became_peered":"2023-12-16T21:57:42.512449+0000","last_unstale":"2023-12-16T21:57:43.462961+0000","last_undegraded":"2023-12-16T21:57:43.462961+0000","last_fullsized":"2023-12-16T21:57:43.462961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:55:25.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869573+0000","last_change":"2023-12-16T21:57:43.486971+0000","last_active":"2023-12-16T21:57:44.869573+0000","last_peered":"2023-12-16T21:57:44.869573+0000","last_clean":"2023-12-16T21:57:44.869573+0000","last_became_active":"2023-12-16T21:57:43.483854+0000","last_became_peered":"2023-12-16T21:57:43.483854+0000","last_unstale":"2023-12-16T21:57:44.869573+0000","last_undegraded":"2023-12-16T21:57:44.869573+0000","last_fullsized":"2023-12-16T21:57:44.869573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263608+0000","last_change":"2023-12-16T21:57:42.501969+0000","last_active":"2023-12-16T21:57:45.263608+0000","last_peered":"2023-12-16T21:57:45.263608+0000","last_clean":"2023-12-16T21:57:45.263608+0000","last_became_active":"2023-12-16T21:57:42.501059+0000","last_became_peered":"2023-12-16T21:57:42.501059+0000","last_unstale":"2023-12-16T21:57:45.263608+0000","last_undegraded":"2023-12-16T21:57:45.263608+0000","last_fullsized":"2023-12-16T21:57:45.263608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:09.342194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459101+0000","last_change":"2023-12-16T21:57:43.506507+0000","last_active":"2023-12-16T21:57:44.459101+0000","last_peered":"2023-12-16T21:57:44.459101+0000","last_clean":"2023-12-16T21:57:44.459101+0000","last_became_active":"2023-12-16T21:57:43.505685+0000","last_became_peered":"2023-12-16T21:57:43.505685+0000","last_unstale":"2023-12-16T21:57:44.459101+0000","last_undegraded":"2023-12-16T21:57:44.459101+0000","last_fullsized":"2023-12-16T21:57:44.459101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:40.013144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869952+0000","last_change":"2023-12-16T21:57:42.503616+0000","last_active":"2023-12-16T21:57:44.869952+0000","last_peered":"2023-12-16T21:57:44.869952+0000","last_clean":"2023-12-16T21:57:44.869952+0000","last_became_active":"2023-12-16T21:57:42.502114+0000","last_became_peered":"2023-12-16T21:57:42.502114+0000","last_unstale":"2023-12-16T21:57:44.869952+0000","last_undegraded":"2023-12-16T21:57:44.869952+0000","last_fullsized":"2023-12-16T21:57:44.869952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:57.160385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262539+0000","last_change":"2023-12-16T21:57:43.500130+0000","last_active":"2023-12-16T21:57:45.262539+0000","last_peered":"2023-12-16T21:57:45.262539+0000","last_clean":"2023-12-16T21:57:45.262539+0000","last_became_active":"2023-12-16T21:57:43.499128+0000","last_became_peered":"2023-12-16T21:57:43.499128+0000","last_unstale":"2023-12-16T21:57:45.262539+0000","last_undegraded":"2023-12-16T21:57:45.262539+0000","last_fullsized":"2023-12-16T21:57:45.262539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:20.834561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.265035+0000","last_change":"2023-12-16T21:57:42.526337+0000","last_active":"2023-12-16T21:57:45.265035+0000","last_peered":"2023-12-16T21:57:45.265035+0000","last_clean":"2023-12-16T21:57:45.265035+0000","last_became_active":"2023-12-16T21:57:42.525796+0000","last_became_peered":"2023-12-16T21:57:42.525796+0000","last_unstale":"2023-12-16T21:57:45.265035+0000","last_undegraded":"2023-12-16T21:57:45.265035+0000","last_fullsized":"2023-12-16T21:57:45.265035+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:00:29.774694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870185+0000","last_change":"2023-12-16T21:57:43.477509+0000","last_active":"2023-12-16T21:57:44.870185+0000","last_peered":"2023-12-16T21:57:44.870185+0000","last_clean":"2023-12-16T21:57:44.870185+0000","last_became_active":"2023-12-16T21:57:43.476854+0000","last_became_peered":"2023-12-16T21:57:43.476854+0000","last_unstale":"2023-12-16T21:57:44.870185+0000","last_undegraded":"2023-12-16T21:57:44.870185+0000","last_fullsized":"2023-12-16T21:57:44.870185+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870093+0000","last_change":"2023-12-16T21:57:42.499802+0000","last_active":"2023-12-16T21:57:44.870093+0000","last_peered":"2023-12-16T21:57:44.870093+0000","last_clean":"2023-12-16T21:57:44.870093+0000","last_became_active":"2023-12-16T21:57:42.498817+0000","last_became_peered":"2023-12-16T21:57:42.498817+0000","last_unstale":"2023-12-16T21:57:44.870093+0000","last_undegraded":"2023-12-16T21:57:44.870093+0000","last_fullsized":"2023-12-16T21:57:44.870093+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263239+0000","last_change":"2023-12-16T21:57:43.490695+0000","last_active":"2023-12-16T21:57:45.263239+0000","last_peered":"2023-12-16T21:57:45.263239+0000","last_clean":"2023-12-16T21:57:45.263239+0000","last_became_active":"2023-12-16T21:57:43.489927+0000","last_became_peered":"2023-12-16T21:57:43.489927+0000","last_unstale":"2023-12-16T21:57:45.263239+0000","last_undegraded":"2023-12-16T21:57:45.263239+0000","last_fullsized":"2023-12-16T21:57:45.263239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458559+0000","last_change":"2023-12-16T21:57:42.523142+0000","last_active":"2023-12-16T21:57:44.458559+0000","last_peered":"2023-12-16T21:57:44.458559+0000","last_clean":"2023-12-16T21:57:44.458559+0000","last_became_active":"2023-12-16T21:57:42.521597+0000","last_became_peered":"2023-12-16T21:57:42.521597+0000","last_unstale":"2023-12-16T21:57:44.458559+0000","last_undegraded":"2023-12-16T21:57:44.458559+0000","last_fullsized":"2023-12-16T21:57:44.458559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252927+0000","last_change":"2023-12-16T21:57:43.483454+0000","last_active":"2023-12-16T21:57:45.252927+0000","last_peered":"2023-12-16T21:57:45.252927+0000","last_clean":"2023-12-16T21:57:45.252927+0000","last_became_active":"2023-12-16T21:57:43.482451+0000","last_became_peered":"2023-12-16T21:57:43.482451+0000","last_unstale":"2023-12-16T21:57:45.252927+0000","last_undegraded":"2023-12-16T21:57:45.252927+0000","last_fullsized":"2023-12-16T21:57:45.252927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261795+0000","last_change":"2023-12-16T21:57:42.522091+0000","last_active":"2023-12-16T21:57:45.261795+0000","last_peered":"2023-12-16T21:57:45.261795+0000","last_clean":"2023-12-16T21:57:45.261795+0000","last_became_active":"2023-12-16T21:57:42.521328+0000","last_became_peered":"2023-12-16T21:57:42.521328+0000","last_unstale":"2023-12-16T21:57:45.261795+0000","last_undegraded":"2023-12-16T21:57:45.261795+0000","last_fullsized":"2023-12-16T21:57:45.261795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457982+0000","last_change":"2023-12-16T21:57:43.494064+0000","last_active":"2023-12-16T21:57:44.457982+0000","last_peered":"2023-12-16T21:57:44.457982+0000","last_clean":"2023-12-16T21:57:44.457982+0000","last_became_active":"2023-12-16T21:57:43.493374+0000","last_became_peered":"2023-12-16T21:57:43.493374+0000","last_unstale":"2023-12-16T21:57:44.457982+0000","last_undegraded":"2023-12-16T21:57:44.457982+0000","last_fullsized":"2023-12-16T21:57:44.457982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:09.494346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869828+0000","last_change":"2023-12-16T21:57:42.508931+0000","last_active":"2023-12-16T21:57:44.869828+0000","last_peered":"2023-12-16T21:57:44.869828+0000","last_clean":"2023-12-16T21:57:44.869828+0000","last_became_active":"2023-12-16T21:57:42.508438+0000","last_became_peered":"2023-12-16T21:57:42.508438+0000","last_unstale":"2023-12-16T21:57:44.869828+0000","last_undegraded":"2023-12-16T21:57:44.869828+0000","last_fullsized":"2023-12-16T21:57:44.869828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:39.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262488+0000","last_change":"2023-12-16T21:57:43.485012+0000","last_active":"2023-12-16T21:57:45.262488+0000","last_peered":"2023-12-16T21:57:45.262488+0000","last_clean":"2023-12-16T21:57:45.262488+0000","last_became_active":"2023-12-16T21:57:43.484177+0000","last_became_peered":"2023-12-16T21:57:43.484177+0000","last_unstale":"2023-12-16T21:57:45.262488+0000","last_undegraded":"2023-12-16T21:57:45.262488+0000","last_fullsized":"2023-12-16T21:57:45.262488+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869858+0000","last_change":"2023-12-16T21:57:42.504338+0000","last_active":"2023-12-16T21:57:44.869858+0000","last_peered":"2023-12-16T21:57:44.869858+0000","last_clean":"2023-12-16T21:57:44.869858+0000","last_became_active":"2023-12-16T21:57:42.503125+0000","last_became_peered":"2023-12-16T21:57:42.503125+0000","last_unstale":"2023-12-16T21:57:44.869858+0000","last_undegraded":"2023-12-16T21:57:44.869858+0000","last_fullsized":"2023-12-16T21:57:44.869858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:23:06.394772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262761+0000","last_change":"2023-12-16T21:57:43.499585+0000","last_active":"2023-12-16T21:57:45.262761+0000","last_peered":"2023-12-16T21:57:45.262761+0000","last_clean":"2023-12-16T21:57:45.262761+0000","last_became_active":"2023-12-16T21:57:43.498790+0000","last_became_peered":"2023-12-16T21:57:43.498790+0000","last_unstale":"2023-12-16T21:57:45.262761+0000","last_undegraded":"2023-12-16T21:57:45.262761+0000","last_fullsized":"2023-12-16T21:57:45.262761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264439+0000","last_change":"2023-12-16T21:57:42.522249+0000","last_active":"2023-12-16T21:57:45.264439+0000","last_peered":"2023-12-16T21:57:45.264439+0000","last_clean":"2023-12-16T21:57:45.264439+0000","last_became_active":"2023-12-16T21:57:42.521585+0000","last_became_peered":"2023-12-16T21:57:42.521585+0000","last_unstale":"2023-12-16T21:57:45.264439+0000","last_undegraded":"2023-12-16T21:57:45.264439+0000","last_fullsized":"2023-12-16T21:57:45.264439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:22.825415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457737+0000","last_change":"2023-12-16T21:57:43.495929+0000","last_active":"2023-12-16T21:57:44.457737+0000","last_peered":"2023-12-16T21:57:44.457737+0000","last_clean":"2023-12-16T21:57:44.457737+0000","last_became_active":"2023-12-16T21:57:43.494696+0000","last_became_peered":"2023-12-16T21:57:43.494696+0000","last_unstale":"2023-12-16T21:57:44.457737+0000","last_undegraded":"2023-12-16T21:57:44.457737+0000","last_fullsized":"2023-12-16T21:57:44.457737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262280+0000","last_change":"2023-12-16T21:57:43.496545+0000","last_active":"2023-12-16T21:57:45.262280+0000","last_peered":"2023-12-16T21:57:45.262280+0000","last_clean":"2023-12-16T21:57:45.262280+0000","last_became_active":"2023-12-16T21:57:43.495846+0000","last_became_peered":"2023-12-16T21:57:43.495846+0000","last_unstale":"2023-12-16T21:57:45.262280+0000","last_undegraded":"2023-12-16T21:57:45.262280+0000","last_fullsized":"2023-12-16T21:57:45.262280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264212+0000","last_change":"2023-12-16T21:57:42.524077+0000","last_active":"2023-12-16T21:57:45.264212+0000","last_peered":"2023-12-16T21:57:45.264212+0000","last_clean":"2023-12-16T21:57:45.264212+0000","last_became_active":"2023-12-16T21:57:42.523590+0000","last_became_peered":"2023-12-16T21:57:42.523590+0000","last_unstale":"2023-12-16T21:57:45.264212+0000","last_undegraded":"2023-12-16T21:57:45.264212+0000","last_fullsized":"2023-12-16T21:57:45.264212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:57.160385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262278+0000","last_change":"2023-12-16T21:57:43.508947+0000","last_active":"2023-12-16T21:57:45.262278+0000","last_peered":"2023-12-16T21:57:45.262278+0000","last_clean":"2023-12-16T21:57:45.262278+0000","last_became_active":"2023-12-16T21:57:43.508424+0000","last_became_peered":"2023-12-16T21:57:43.508424+0000","last_unstale":"2023-12-16T21:57:45.262278+0000","last_undegraded":"2023-12-16T21:57:45.262278+0000","last_fullsized":"2023-12-16T21:57:45.262278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264760+0000","last_change":"2023-12-16T21:57:42.522477+0000","last_active":"2023-12-16T21:57:45.264760+0000","last_peered":"2023-12-16T21:57:45.264760+0000","last_clean":"2023-12-16T21:57:45.264760+0000","last_became_active":"2023-12-16T21:57:42.521732+0000","last_became_peered":"2023-12-16T21:57:42.521732+0000","last_unstale":"2023-12-16T21:57:45.264760+0000","last_undegraded":"2023-12-16T21:57:45.264760+0000","last_fullsized":"2023-12-16T21:57:45.264760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252926+0000","last_change":"2023-12-16T21:57:43.481470+0000","last_active":"2023-12-16T21:57:45.252926+0000","last_peered":"2023-12-16T21:57:45.252926+0000","last_clean":"2023-12-16T21:57:45.252926+0000","last_became_active":"2023-12-16T21:57:43.480541+0000","last_became_peered":"2023-12-16T21:57:43.480541+0000","last_unstale":"2023-12-16T21:57:45.252926+0000","last_undegraded":"2023-12-16T21:57:45.252926+0000","last_fullsized":"2023-12-16T21:57:45.252926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263338+0000","last_change":"2023-12-16T21:57:42.512776+0000","last_active":"2023-12-16T21:57:45.263338+0000","last_peered":"2023-12-16T21:57:45.263338+0000","last_clean":"2023-12-16T21:57:45.263338+0000","last_became_active":"2023-12-16T21:57:42.512393+0000","last_became_peered":"2023-12-16T21:57:42.512393+0000","last_unstale":"2023-12-16T21:57:45.263338+0000","last_undegraded":"2023-12-16T21:57:45.263338+0000","last_fullsized":"2023-12-16T21:57:45.263338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263375+0000","last_change":"2023-12-16T21:57:43.499571+0000","last_active":"2023-12-16T21:57:45.263375+0000","last_peered":"2023-12-16T21:57:45.263375+0000","last_clean":"2023-12-16T21:57:45.263375+0000","last_became_active":"2023-12-16T21:57:43.498880+0000","last_became_peered":"2023-12-16T21:57:43.498880+0000","last_unstale":"2023-12-16T21:57:45.263375+0000","last_undegraded":"2023-12-16T21:57:45.263375+0000","last_fullsized":"2023-12-16T21:57:45.263375+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.253109+0000","last_change":"2023-12-16T21:57:42.522619+0000","last_active":"2023-12-16T21:57:45.253109+0000","last_peered":"2023-12-16T21:57:45.253109+0000","last_clean":"2023-12-16T21:57:45.253109+0000","last_became_active":"2023-12-16T21:57:42.522259+0000","last_became_peered":"2023-12-16T21:57:42.522259+0000","last_unstale":"2023-12-16T21:57:45.253109+0000","last_undegraded":"2023-12-16T21:57:45.253109+0000","last_fullsized":"2023-12-16T21:57:45.253109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476742,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":827392,"data_stored":670753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]}]},{"osd":6,"up_from":16,"seq":68719476741,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.974}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1820000000000004}]}]},{"osd":5,"up_from":16,"seq":68719476742,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":79893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]}]},{"osd":4,"up_from":16,"seq":68719476740,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":79929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]}]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":75115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":237568,"data_stored":79963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":823296,"data_stored":664509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":73503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:57:47.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:57:47.697 INFO:tasks.ceph.mds.d.smithi122.stderr:2023-12-16T21:57:47.695+0000 7f3d98f9f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:57:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b4632640 1 Processor -- start 2023-12-16T21:57:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b4632640 1 -- start start 2023-12-16T21:57:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b4632640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f52ac0b4720 0x7f52ac0d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b4632640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac15e560 con 0x7f52ac15e140 2023-12-16T21:57:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b4632640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ac160f20 con 0x7f52ac0d1930 2023-12-16T21:57:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b4632640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ac0d1de0 con 0x7f52ac0b4720 2023-12-16T21:57:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b23ce640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f52ac0b4720 0x7f52ac0d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b2bcf640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f52ac15e140 0x7f52ac141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51124/0 (socket says 172.21.15.28:51124) 2023-12-16T21:57:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b2bcf640 1 -- 172.21.15.28:0/3376454681 learned_addr learned my addr 172.21.15.28:0/3376454681 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b23ce640 1 -- 172.21.15.28:0/3376454681 >> v1:172.21.15.122:6789/0 conn(0x7f52ac0d1930 legacy=0x7f52ac15da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b23ce640 1 -- 172.21.15.28:0/3376454681 >> v1:172.21.15.28:6789/0 conn(0x7f52ac15e140 legacy=0x7f52ac141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.740+0000 7f52b23ce640 1 -- 172.21.15.28:0/3376454681 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ac161200 con 0x7f52ac0b4720 2023-12-16T21:57:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b23ce640 1 --2- 172.21.15.28:0/3376454681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0b4720 0x7f52ac0d13f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52a80097c0 tx=0x7f52a801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b1d5acdbc0566c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b13cc640 1 -- 172.21.15.28:0/3376454681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8004230 con 0x7f52ac0b4720 2023-12-16T21:57:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b13cc640 1 -- 172.21.15.28:0/3376454681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a80a7050 con 0x7f52ac0b4720 2023-12-16T21:57:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b13cc640 1 -- 172.21.15.28:0/3376454681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8003620 con 0x7f52ac0b4720 2023-12-16T21:57:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- 172.21.15.28:0/3376454681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0b4720 msgr2=0x7f52ac0d13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 --2- 172.21.15.28:0/3376454681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0b4720 0x7f52ac0d13f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52a80097c0 tx=0x7f52a801c440 comp rx=0 tx=0).stop 2023-12-16T21:57:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- 172.21.15.28:0/3376454681 shutdown_connections 2023-12-16T21:57:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 --2- 172.21.15.28:0/3376454681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0b4720 0x7f52ac0d13f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- 172.21.15.28:0/3376454681 >> 172.21.15.28:0/3376454681 conn(0x7f52ac06a930 msgr2=0x7f52ac06ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- 172.21.15.28:0/3376454681 shutdown_connections 2023-12-16T21:57:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- 172.21.15.28:0/3376454681 wait complete. 2023-12-16T21:57:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 Processor -- start 2023-12-16T21:57:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- start start 2023-12-16T21:57:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52ac0b4720 0x7f52ac154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0d1930 0x7f52ac139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f52ac15e140 0x7f52ac154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac142bc0 con 0x7f52ac0d1930 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac13a9a0 con 0x7f52ac0b4720 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b4632640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ac13ac80 con 0x7f52ac15e140 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0d1930 0x7f52ac139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0d1930 0x7f52ac139200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45844/0 (socket says 172.21.15.28:45844) 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 -- 172.21.15.28:0/3345521652 learned_addr learned my addr 172.21.15.28:0/3345521652 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b23ce640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52ac0b4720 0x7f52ac154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b2bcf640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f52ac15e140 0x7f52ac154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 -- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f52ac15e140 msgr2=0x7f52ac154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f52ac15e140 0x7f52ac154dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 -- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52ac0b4720 msgr2=0x7f52ac154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52ac0b4720 0x7f52ac154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 -- 172.21.15.28:0/3345521652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a8009420 con 0x7f52ac0d1930 2023-12-16T21:57:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b23ce640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52ac0b4720 0x7f52ac154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.744+0000 7f52b1bcd640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0d1930 0x7f52ac139200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c7a0 tx=0x7f52a000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f529effd640 1 -- 172.21.15.28:0/3345521652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0003d90 con 0x7f52ac0d1930 2023-12-16T21:57:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f529effd640 1 -- 172.21.15.28:0/3345521652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a0005070 con 0x7f52ac0d1930 2023-12-16T21:57:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ac13af60 con 0x7f52ac0d1930 2023-12-16T21:57:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f529effd640 1 -- 172.21.15.28:0/3345521652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0005460 con 0x7f52ac0d1930 2023-12-16T21:57:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ac155310 con 0x7f52ac0d1930 2023-12-16T21:57:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ac0674d0 con 0x7f52ac0d1930 2023-12-16T21:57:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f529effd640 1 -- 172.21.15.28:0/3345521652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f52a00055c0 con 0x7f52ac0d1930 2023-12-16T21:57:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f529effd640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f52900664a0 0x7f5290068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f529effd640 1 -- 172.21.15.28:0/3345521652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52a0083300 con 0x7f52ac0d1930 2023-12-16T21:57:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f52b23ce640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f52900664a0 0x7f5290068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.748+0000 7f52b23ce640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f52900664a0 0x7f5290068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f52a80ad000 tx=0x7f52a8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:47.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.752+0000 7f529effd640 1 -- 172.21.15.28:0/3345521652 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f52a0050780 con 0x7f52ac0d1930 2023-12-16T21:57:47.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.908+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52ac0b4b00 con 0x7f52900664a0 2023-12-16T21:57:47.928 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.928+0000 7f529effd640 1 -- 172.21.15.28:0/3345521652 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264495 (secure 0 0 0) 0x7f52a8004470 con 0x7f52900664a0 2023-12-16T21:57:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f52900664a0 msgr2=0x7f5290068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f52900664a0 0x7f5290068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f52a80ad000 tx=0x7f52a8008040 comp rx=0 tx=0).stop 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0d1930 msgr2=0x7f52ac139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0d1930 0x7f52ac139200 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52a000c7a0 tx=0x7f52a000cc60 comp rx=0 tx=0).stop 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 shutdown_connections 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f52900664a0 0x7f5290068960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f52ac15e140 0x7f52ac154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac0d1930 0x7f52ac139200 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 --2- 172.21.15.28:0/3345521652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f52ac0b4720 0x7f52ac154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:47.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 >> 172.21.15.28:0/3345521652 conn(0x7f52ac06a930 msgr2=0x7f52ac06c410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 shutdown_connections 2023-12-16T21:57:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:47.932+0000 7f52b4632640 1 -- 172.21.15.28:0/3345521652 wait complete. 2023-12-16T21:57:47.949 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-12-16T21:57:47.382660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235776,"kb_used_data":2980,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738944,"statfs":{"total":773094113280,"available":772852678656,"internally_reserved":0,"allocated":3051520,"data_stored":1805080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":60,"apply_latency_ms":60,"commit_latency_ns":60000000,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.918541"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264489+0000","last_change":"2023-12-16T21:57:43.518611+0000","last_active":"2023-12-16T21:57:45.264489+0000","last_peered":"2023-12-16T21:57:45.264489+0000","last_clean":"2023-12-16T21:57:45.264489+0000","last_became_active":"2023-12-16T21:57:43.517704+0000","last_became_peered":"2023-12-16T21:57:43.517704+0000","last_unstale":"2023-12-16T21:57:45.264489+0000","last_undegraded":"2023-12-16T21:57:45.264489+0000","last_fullsized":"2023-12-16T21:57:45.264489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:49:13.751852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870153+0000","last_change":"2023-12-16T21:57:43.485837+0000","last_active":"2023-12-16T21:57:44.870153+0000","last_peered":"2023-12-16T21:57:44.870153+0000","last_clean":"2023-12-16T21:57:44.870153+0000","last_became_active":"2023-12-16T21:57:43.484881+0000","last_became_peered":"2023-12-16T21:57:43.484881+0000","last_unstale":"2023-12-16T21:57:44.870153+0000","last_undegraded":"2023-12-16T21:57:44.870153+0000","last_fullsized":"2023-12-16T21:57:44.870153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252805+0000","last_change":"2023-12-16T21:57:43.492016+0000","last_active":"2023-12-16T21:57:45.252805+0000","last_peered":"2023-12-16T21:57:45.252805+0000","last_clean":"2023-12-16T21:57:45.252805+0000","last_became_active":"2023-12-16T21:57:43.491114+0000","last_became_peered":"2023-12-16T21:57:43.491114+0000","last_unstale":"2023-12-16T21:57:45.252805+0000","last_undegraded":"2023-12-16T21:57:45.252805+0000","last_fullsized":"2023-12-16T21:57:45.252805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262086+0000","last_change":"2023-12-16T21:57:43.510356+0000","last_active":"2023-12-16T21:57:45.262086+0000","last_peered":"2023-12-16T21:57:45.262086+0000","last_clean":"2023-12-16T21:57:45.262086+0000","last_became_active":"2023-12-16T21:57:43.509819+0000","last_became_peered":"2023-12-16T21:57:43.509819+0000","last_unstale":"2023-12-16T21:57:45.262086+0000","last_undegraded":"2023-12-16T21:57:45.262086+0000","last_fullsized":"2023-12-16T21:57:45.262086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264164+0000","last_change":"2023-12-16T21:57:43.507126+0000","last_active":"2023-12-16T21:57:45.264164+0000","last_peered":"2023-12-16T21:57:45.264164+0000","last_clean":"2023-12-16T21:57:45.264164+0000","last_became_active":"2023-12-16T21:57:43.506352+0000","last_became_peered":"2023-12-16T21:57:43.506352+0000","last_unstale":"2023-12-16T21:57:45.264164+0000","last_undegraded":"2023-12-16T21:57:45.264164+0000","last_fullsized":"2023-12-16T21:57:45.264164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:04:09.033093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262865+0000","last_change":"2023-12-16T21:57:43.483088+0000","last_active":"2023-12-16T21:57:45.262865+0000","last_peered":"2023-12-16T21:57:45.262865+0000","last_clean":"2023-12-16T21:57:45.262865+0000","last_became_active":"2023-12-16T21:57:43.482278+0000","last_became_peered":"2023-12-16T21:57:43.482278+0000","last_unstale":"2023-12-16T21:57:45.262865+0000","last_undegraded":"2023-12-16T21:57:45.262865+0000","last_fullsized":"2023-12-16T21:57:45.262865+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:15.360923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261922+0000","last_change":"2023-12-16T21:57:43.494598+0000","last_active":"2023-12-16T21:57:45.261922+0000","last_peered":"2023-12-16T21:57:45.261922+0000","last_clean":"2023-12-16T21:57:45.261922+0000","last_became_active":"2023-12-16T21:57:43.493867+0000","last_became_peered":"2023-12-16T21:57:43.493867+0000","last_unstale":"2023-12-16T21:57:45.261922+0000","last_undegraded":"2023-12-16T21:57:45.261922+0000","last_fullsized":"2023-12-16T21:57:45.261922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262085+0000","last_change":"2023-12-16T21:57:43.504864+0000","last_active":"2023-12-16T21:57:45.262085+0000","last_peered":"2023-12-16T21:57:45.262085+0000","last_clean":"2023-12-16T21:57:45.262085+0000","last_became_active":"2023-12-16T21:57:43.504140+0000","last_became_peered":"2023-12-16T21:57:43.504140+0000","last_unstale":"2023-12-16T21:57:45.262085+0000","last_undegraded":"2023-12-16T21:57:45.262085+0000","last_fullsized":"2023-12-16T21:57:45.262085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:22.190906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264734+0000","last_change":"2023-12-16T21:57:43.518225+0000","last_active":"2023-12-16T21:57:45.264734+0000","last_peered":"2023-12-16T21:57:45.264734+0000","last_clean":"2023-12-16T21:57:45.264734+0000","last_became_active":"2023-12-16T21:57:43.517602+0000","last_became_peered":"2023-12-16T21:57:43.517602+0000","last_unstale":"2023-12-16T21:57:45.264734+0000","last_undegraded":"2023-12-16T21:57:45.264734+0000","last_fullsized":"2023-12-16T21:57:45.264734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:46:04.514997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.460177+0000","last_change":"2023-12-16T21:57:43.468564+0000","last_active":"2023-12-16T21:57:44.460177+0000","last_peered":"2023-12-16T21:57:44.460177+0000","last_clean":"2023-12-16T21:57:44.460177+0000","last_became_active":"2023-12-16T21:57:43.467674+0000","last_became_peered":"2023-12-16T21:57:43.467674+0000","last_unstale":"2023-12-16T21:57:44.460177+0000","last_undegraded":"2023-12-16T21:57:44.460177+0000","last_fullsized":"2023-12-16T21:57:44.460177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458605+0000","last_change":"2023-12-16T21:57:43.474153+0000","last_active":"2023-12-16T21:57:44.458605+0000","last_peered":"2023-12-16T21:57:44.458605+0000","last_clean":"2023-12-16T21:57:44.458605+0000","last_became_active":"2023-12-16T21:57:43.467788+0000","last_became_peered":"2023-12-16T21:57:43.467788+0000","last_unstale":"2023-12-16T21:57:44.458605+0000","last_undegraded":"2023-12-16T21:57:44.458605+0000","last_fullsized":"2023-12-16T21:57:44.458605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458900+0000","last_change":"2023-12-16T21:57:43.494364+0000","last_active":"2023-12-16T21:57:44.458900+0000","last_peered":"2023-12-16T21:57:44.458900+0000","last_clean":"2023-12-16T21:57:44.458900+0000","last_became_active":"2023-12-16T21:57:43.491952+0000","last_became_peered":"2023-12-16T21:57:43.491952+0000","last_unstale":"2023-12-16T21:57:44.458900+0000","last_undegraded":"2023-12-16T21:57:44.458900+0000","last_fullsized":"2023-12-16T21:57:44.458900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:16.932172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262657+0000","last_change":"2023-12-16T21:57:43.490348+0000","last_active":"2023-12-16T21:57:45.262657+0000","last_peered":"2023-12-16T21:57:45.262657+0000","last_clean":"2023-12-16T21:57:45.262657+0000","last_became_active":"2023-12-16T21:57:43.489582+0000","last_became_peered":"2023-12-16T21:57:43.489582+0000","last_unstale":"2023-12-16T21:57:45.262657+0000","last_undegraded":"2023-12-16T21:57:45.262657+0000","last_fullsized":"2023-12-16T21:57:45.262657+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252238+0000","last_change":"2023-12-16T21:57:43.481446+0000","last_active":"2023-12-16T21:57:45.252238+0000","last_peered":"2023-12-16T21:57:45.252238+0000","last_clean":"2023-12-16T21:57:45.252238+0000","last_became_active":"2023-12-16T21:57:43.480422+0000","last_became_peered":"2023-12-16T21:57:43.480422+0000","last_unstale":"2023-12-16T21:57:45.252238+0000","last_undegraded":"2023-12-16T21:57:45.252238+0000","last_fullsized":"2023-12-16T21:57:45.252238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869559+0000","last_change":"2023-12-16T21:57:43.507792+0000","last_active":"2023-12-16T21:57:44.869559+0000","last_peered":"2023-12-16T21:57:44.869559+0000","last_clean":"2023-12-16T21:57:44.869559+0000","last_became_active":"2023-12-16T21:57:43.507030+0000","last_became_peered":"2023-12-16T21:57:43.507030+0000","last_unstale":"2023-12-16T21:57:44.869559+0000","last_undegraded":"2023-12-16T21:57:44.869559+0000","last_fullsized":"2023-12-16T21:57:44.869559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261659+0000","last_change":"2023-12-16T21:57:43.485531+0000","last_active":"2023-12-16T21:57:45.261659+0000","last_peered":"2023-12-16T21:57:45.261659+0000","last_clean":"2023-12-16T21:57:45.261659+0000","last_became_active":"2023-12-16T21:57:43.482538+0000","last_became_peered":"2023-12-16T21:57:43.482538+0000","last_unstale":"2023-12-16T21:57:45.261659+0000","last_undegraded":"2023-12-16T21:57:45.261659+0000","last_fullsized":"2023-12-16T21:57:45.261659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263056+0000","last_change":"2023-12-16T21:57:43.507572+0000","last_active":"2023-12-16T21:57:45.263056+0000","last_peered":"2023-12-16T21:57:45.263056+0000","last_clean":"2023-12-16T21:57:45.263056+0000","last_became_active":"2023-12-16T21:57:43.507072+0000","last_became_peered":"2023-12-16T21:57:43.507072+0000","last_unstale":"2023-12-16T21:57:45.263056+0000","last_undegraded":"2023-12-16T21:57:45.263056+0000","last_fullsized":"2023-12-16T21:57:45.263056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:49:13.751852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.502007+0000","last_change":"2023-12-16T21:57:43.487657+0000","last_active":"2023-12-16T21:57:43.502007+0000","last_peered":"2023-12-16T21:57:43.502007+0000","last_clean":"2023-12-16T21:57:43.502007+0000","last_became_active":"2023-12-16T21:57:43.484848+0000","last_became_peered":"2023-12-16T21:57:43.484848+0000","last_unstale":"2023-12-16T21:57:43.502007+0000","last_undegraded":"2023-12-16T21:57:43.502007+0000","last_fullsized":"2023-12-16T21:57:43.502007+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:30:24.150158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264990+0000","last_change":"2023-12-16T21:57:43.503136+0000","last_active":"2023-12-16T21:57:45.264990+0000","last_peered":"2023-12-16T21:57:45.264990+0000","last_clean":"2023-12-16T21:57:45.264990+0000","last_became_active":"2023-12-16T21:57:43.502321+0000","last_became_peered":"2023-12-16T21:57:43.502321+0000","last_unstale":"2023-12-16T21:57:45.264990+0000","last_undegraded":"2023-12-16T21:57:45.264990+0000","last_fullsized":"2023-12-16T21:57:45.264990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:29.457727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458117+0000","last_change":"2023-12-16T21:57:43.496151+0000","last_active":"2023-12-16T21:57:44.458117+0000","last_peered":"2023-12-16T21:57:44.458117+0000","last_clean":"2023-12-16T21:57:44.458117+0000","last_became_active":"2023-12-16T21:57:43.495293+0000","last_became_peered":"2023-12-16T21:57:43.495293+0000","last_unstale":"2023-12-16T21:57:44.458117+0000","last_undegraded":"2023-12-16T21:57:44.458117+0000","last_fullsized":"2023-12-16T21:57:44.458117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252271+0000","last_change":"2023-12-16T21:57:43.507839+0000","last_active":"2023-12-16T21:57:45.252271+0000","last_peered":"2023-12-16T21:57:45.252271+0000","last_clean":"2023-12-16T21:57:45.252271+0000","last_became_active":"2023-12-16T21:57:43.507150+0000","last_became_peered":"2023-12-16T21:57:43.507150+0000","last_unstale":"2023-12-16T21:57:45.252271+0000","last_undegraded":"2023-12-16T21:57:45.252271+0000","last_fullsized":"2023-12-16T21:57:45.252271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263163+0000","last_change":"2023-12-16T21:57:43.487735+0000","last_active":"2023-12-16T21:57:45.263163+0000","last_peered":"2023-12-16T21:57:45.263163+0000","last_clean":"2023-12-16T21:57:45.263163+0000","last_became_active":"2023-12-16T21:57:43.486972+0000","last_became_peered":"2023-12-16T21:57:43.486972+0000","last_unstale":"2023-12-16T21:57:45.263163+0000","last_undegraded":"2023-12-16T21:57:45.263163+0000","last_fullsized":"2023-12-16T21:57:45.263163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:46:04.514997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262348+0000","last_change":"2023-12-16T21:57:43.513648+0000","last_active":"2023-12-16T21:57:45.262348+0000","last_peered":"2023-12-16T21:57:45.262348+0000","last_clean":"2023-12-16T21:57:45.262348+0000","last_became_active":"2023-12-16T21:57:43.513086+0000","last_became_peered":"2023-12-16T21:57:43.513086+0000","last_unstale":"2023-12-16T21:57:45.262348+0000","last_undegraded":"2023-12-16T21:57:45.262348+0000","last_fullsized":"2023-12-16T21:57:45.262348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261798+0000","last_change":"2023-12-16T21:57:43.505290+0000","last_active":"2023-12-16T21:57:45.261798+0000","last_peered":"2023-12-16T21:57:45.261798+0000","last_clean":"2023-12-16T21:57:45.261798+0000","last_became_active":"2023-12-16T21:57:43.504600+0000","last_became_peered":"2023-12-16T21:57:43.504600+0000","last_unstale":"2023-12-16T21:57:45.261798+0000","last_undegraded":"2023-12-16T21:57:45.261798+0000","last_fullsized":"2023-12-16T21:57:45.261798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263041+0000","last_change":"2023-12-16T21:57:43.510116+0000","last_active":"2023-12-16T21:57:45.263041+0000","last_peered":"2023-12-16T21:57:45.263041+0000","last_clean":"2023-12-16T21:57:45.263041+0000","last_became_active":"2023-12-16T21:57:43.509260+0000","last_became_peered":"2023-12-16T21:57:43.509260+0000","last_unstale":"2023-12-16T21:57:45.263041+0000","last_undegraded":"2023-12-16T21:57:45.263041+0000","last_fullsized":"2023-12-16T21:57:45.263041+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.460344+0000","last_change":"2023-12-16T21:57:43.489237+0000","last_active":"2023-12-16T21:57:44.460344+0000","last_peered":"2023-12-16T21:57:44.460344+0000","last_clean":"2023-12-16T21:57:44.460344+0000","last_became_active":"2023-12-16T21:57:43.488252+0000","last_became_peered":"2023-12-16T21:57:43.488252+0000","last_unstale":"2023-12-16T21:57:44.460344+0000","last_undegraded":"2023-12-16T21:57:44.460344+0000","last_fullsized":"2023-12-16T21:57:44.460344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869771+0000","last_change":"2023-12-16T21:57:43.502339+0000","last_active":"2023-12-16T21:57:44.869771+0000","last_peered":"2023-12-16T21:57:44.869771+0000","last_clean":"2023-12-16T21:57:44.869771+0000","last_became_active":"2023-12-16T21:57:43.501434+0000","last_became_peered":"2023-12-16T21:57:43.501434+0000","last_unstale":"2023-12-16T21:57:44.869771+0000","last_undegraded":"2023-12-16T21:57:44.869771+0000","last_fullsized":"2023-12-16T21:57:44.869771+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263170+0000","last_change":"2023-12-16T21:57:43.509617+0000","last_active":"2023-12-16T21:57:45.263170+0000","last_peered":"2023-12-16T21:57:45.263170+0000","last_clean":"2023-12-16T21:57:45.263170+0000","last_became_active":"2023-12-16T21:57:43.508922+0000","last_became_peered":"2023-12-16T21:57:43.508922+0000","last_unstale":"2023-12-16T21:57:45.263170+0000","last_undegraded":"2023-12-16T21:57:45.263170+0000","last_fullsized":"2023-12-16T21:57:45.263170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252408+0000","last_change":"2023-12-16T21:57:43.505933+0000","last_active":"2023-12-16T21:57:45.252408+0000","last_peered":"2023-12-16T21:57:45.252408+0000","last_clean":"2023-12-16T21:57:45.252408+0000","last_became_active":"2023-12-16T21:57:43.504463+0000","last_became_peered":"2023-12-16T21:57:43.504463+0000","last_unstale":"2023-12-16T21:57:45.252408+0000","last_undegraded":"2023-12-16T21:57:45.252408+0000","last_fullsized":"2023-12-16T21:57:45.252408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263332+0000","last_change":"2023-12-16T21:57:43.501738+0000","last_active":"2023-12-16T21:57:45.263332+0000","last_peered":"2023-12-16T21:57:45.263332+0000","last_clean":"2023-12-16T21:57:45.263332+0000","last_became_active":"2023-12-16T21:57:43.501014+0000","last_became_peered":"2023-12-16T21:57:43.501014+0000","last_unstale":"2023-12-16T21:57:45.263332+0000","last_undegraded":"2023-12-16T21:57:45.263332+0000","last_fullsized":"2023-12-16T21:57:45.263332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:47:38.932092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.480394+0000","last_change":"2023-12-16T21:57:43.477348+0000","last_active":"2023-12-16T21:57:43.480394+0000","last_peered":"2023-12-16T21:57:43.480394+0000","last_clean":"2023-12-16T21:57:43.480394+0000","last_became_active":"2023-12-16T21:57:43.476616+0000","last_became_peered":"2023-12-16T21:57:43.476616+0000","last_unstale":"2023-12-16T21:57:43.480394+0000","last_undegraded":"2023-12-16T21:57:43.480394+0000","last_fullsized":"2023-12-16T21:57:43.480394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:48:17.536347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457542+0000","last_change":"2023-12-16T21:57:43.483725+0000","last_active":"2023-12-16T21:57:44.457542+0000","last_peered":"2023-12-16T21:57:44.457542+0000","last_clean":"2023-12-16T21:57:44.457542+0000","last_became_active":"2023-12-16T21:57:43.482870+0000","last_became_peered":"2023-12-16T21:57:43.482870+0000","last_unstale":"2023-12-16T21:57:44.457542+0000","last_undegraded":"2023-12-16T21:57:44.457542+0000","last_fullsized":"2023-12-16T21:57:44.457542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458048+0000","last_change":"2023-12-16T21:57:43.492281+0000","last_active":"2023-12-16T21:57:44.458048+0000","last_peered":"2023-12-16T21:57:44.458048+0000","last_clean":"2023-12-16T21:57:44.458048+0000","last_became_active":"2023-12-16T21:57:43.491538+0000","last_became_peered":"2023-12-16T21:57:43.491538+0000","last_unstale":"2023-12-16T21:57:44.458048+0000","last_undegraded":"2023-12-16T21:57:44.458048+0000","last_fullsized":"2023-12-16T21:57:44.458048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:21:57.846367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262966+0000","last_change":"2023-12-16T21:57:42.522074+0000","last_active":"2023-12-16T21:57:45.262966+0000","last_peered":"2023-12-16T21:57:45.262966+0000","last_clean":"2023-12-16T21:57:45.262966+0000","last_became_active":"2023-12-16T21:57:42.521769+0000","last_became_peered":"2023-12-16T21:57:42.521769+0000","last_unstale":"2023-12-16T21:57:45.262966+0000","last_undegraded":"2023-12-16T21:57:45.262966+0000","last_fullsized":"2023-12-16T21:57:45.262966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:42:59.000744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252714+0000","last_change":"2023-12-16T21:57:43.495840+0000","last_active":"2023-12-16T21:57:45.252714+0000","last_peered":"2023-12-16T21:57:45.252714+0000","last_clean":"2023-12-16T21:57:45.252714+0000","last_became_active":"2023-12-16T21:57:43.495114+0000","last_became_peered":"2023-12-16T21:57:43.495114+0000","last_unstale":"2023-12-16T21:57:45.252714+0000","last_undegraded":"2023-12-16T21:57:45.252714+0000","last_fullsized":"2023-12-16T21:57:45.252714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458001+0000","last_change":"2023-12-16T21:57:42.515225+0000","last_active":"2023-12-16T21:57:44.458001+0000","last_peered":"2023-12-16T21:57:44.458001+0000","last_clean":"2023-12-16T21:57:44.458001+0000","last_became_active":"2023-12-16T21:57:42.514509+0000","last_became_peered":"2023-12-16T21:57:42.514509+0000","last_unstale":"2023-12-16T21:57:44.458001+0000","last_undegraded":"2023-12-16T21:57:44.458001+0000","last_fullsized":"2023-12-16T21:57:44.458001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:21.179733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262272+0000","last_change":"2023-12-16T21:57:42.515658+0000","last_active":"2023-12-16T21:57:45.262272+0000","last_peered":"2023-12-16T21:57:45.262272+0000","last_clean":"2023-12-16T21:57:45.262272+0000","last_became_active":"2023-12-16T21:57:42.515275+0000","last_became_peered":"2023-12-16T21:57:42.515275+0000","last_unstale":"2023-12-16T21:57:45.262272+0000","last_undegraded":"2023-12-16T21:57:45.262272+0000","last_fullsized":"2023-12-16T21:57:45.262272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:00:29.774694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262293+0000","last_change":"2023-12-16T21:57:43.493987+0000","last_active":"2023-12-16T21:57:45.262293+0000","last_peered":"2023-12-16T21:57:45.262293+0000","last_clean":"2023-12-16T21:57:45.262293+0000","last_became_active":"2023-12-16T21:57:43.493193+0000","last_became_peered":"2023-12-16T21:57:43.493193+0000","last_unstale":"2023-12-16T21:57:45.262293+0000","last_undegraded":"2023-12-16T21:57:45.262293+0000","last_fullsized":"2023-12-16T21:57:45.262293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261893+0000","last_change":"2023-12-16T21:57:42.508909+0000","last_active":"2023-12-16T21:57:45.261893+0000","last_peered":"2023-12-16T21:57:45.261893+0000","last_clean":"2023-12-16T21:57:45.261893+0000","last_became_active":"2023-12-16T21:57:42.508250+0000","last_became_peered":"2023-12-16T21:57:42.508250+0000","last_unstale":"2023-12-16T21:57:45.261893+0000","last_undegraded":"2023-12-16T21:57:45.261893+0000","last_fullsized":"2023-12-16T21:57:45.261893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:39.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869537+0000","last_change":"2023-12-16T21:57:43.487695+0000","last_active":"2023-12-16T21:57:44.869537+0000","last_peered":"2023-12-16T21:57:44.869537+0000","last_clean":"2023-12-16T21:57:44.869537+0000","last_became_active":"2023-12-16T21:57:43.486936+0000","last_became_peered":"2023-12-16T21:57:43.486936+0000","last_unstale":"2023-12-16T21:57:44.869537+0000","last_undegraded":"2023-12-16T21:57:44.869537+0000","last_fullsized":"2023-12-16T21:57:44.869537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458471+0000","last_change":"2023-12-16T21:57:42.526210+0000","last_active":"2023-12-16T21:57:44.458471+0000","last_peered":"2023-12-16T21:57:44.458471+0000","last_clean":"2023-12-16T21:57:44.458471+0000","last_became_active":"2023-12-16T21:57:42.525460+0000","last_became_peered":"2023-12-16T21:57:42.525460+0000","last_unstale":"2023-12-16T21:57:44.458471+0000","last_undegraded":"2023-12-16T21:57:44.458471+0000","last_fullsized":"2023-12-16T21:57:44.458471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:43:36.373508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263137+0000","last_change":"2023-12-16T21:57:43.483521+0000","last_active":"2023-12-16T21:57:45.263137+0000","last_peered":"2023-12-16T21:57:45.263137+0000","last_clean":"2023-12-16T21:57:45.263137+0000","last_became_active":"2023-12-16T21:57:43.482802+0000","last_became_peered":"2023-12-16T21:57:43.482802+0000","last_unstale":"2023-12-16T21:57:45.263137+0000","last_undegraded":"2023-12-16T21:57:45.263137+0000","last_fullsized":"2023-12-16T21:57:45.263137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:29.175289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263361+0000","last_change":"2023-12-16T21:57:42.517029+0000","last_active":"2023-12-16T21:57:45.263361+0000","last_peered":"2023-12-16T21:57:45.263361+0000","last_clean":"2023-12-16T21:57:45.263361+0000","last_became_active":"2023-12-16T21:57:42.516592+0000","last_became_peered":"2023-12-16T21:57:42.516592+0000","last_unstale":"2023-12-16T21:57:45.263361+0000","last_undegraded":"2023-12-16T21:57:45.263361+0000","last_fullsized":"2023-12-16T21:57:45.263361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:42:59.000744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252990+0000","last_change":"2023-12-16T21:57:43.480932+0000","last_active":"2023-12-16T21:57:45.252990+0000","last_peered":"2023-12-16T21:57:45.252990+0000","last_clean":"2023-12-16T21:57:45.252990+0000","last_became_active":"2023-12-16T21:57:43.480056+0000","last_became_peered":"2023-12-16T21:57:43.480056+0000","last_unstale":"2023-12-16T21:57:45.252990+0000","last_undegraded":"2023-12-16T21:57:45.252990+0000","last_fullsized":"2023-12-16T21:57:45.252990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869739+0000","last_change":"2023-12-16T21:57:42.499117+0000","last_active":"2023-12-16T21:57:44.869739+0000","last_peered":"2023-12-16T21:57:44.869739+0000","last_clean":"2023-12-16T21:57:44.869739+0000","last_became_active":"2023-12-16T21:57:42.498315+0000","last_became_peered":"2023-12-16T21:57:42.498315+0000","last_unstale":"2023-12-16T21:57:44.869739+0000","last_undegraded":"2023-12-16T21:57:44.869739+0000","last_fullsized":"2023-12-16T21:57:44.869739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:14:36.842626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459102+0000","last_change":"2023-12-16T21:57:43.507811+0000","last_active":"2023-12-16T21:57:44.459102+0000","last_peered":"2023-12-16T21:57:44.459102+0000","last_clean":"2023-12-16T21:57:44.459102+0000","last_became_active":"2023-12-16T21:57:43.507089+0000","last_became_peered":"2023-12-16T21:57:43.507089+0000","last_unstale":"2023-12-16T21:57:44.459102+0000","last_undegraded":"2023-12-16T21:57:44.459102+0000","last_fullsized":"2023-12-16T21:57:44.459102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:43:00.011918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458372+0000","last_change":"2023-12-16T21:57:42.512797+0000","last_active":"2023-12-16T21:57:44.458372+0000","last_peered":"2023-12-16T21:57:44.458372+0000","last_clean":"2023-12-16T21:57:44.458372+0000","last_became_active":"2023-12-16T21:57:42.512055+0000","last_became_peered":"2023-12-16T21:57:42.512055+0000","last_unstale":"2023-12-16T21:57:44.458372+0000","last_undegraded":"2023-12-16T21:57:44.458372+0000","last_fullsized":"2023-12-16T21:57:44.458372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:15.920999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264337+0000","last_change":"2023-12-16T21:57:43.510475+0000","last_active":"2023-12-16T21:57:45.264337+0000","last_peered":"2023-12-16T21:57:45.264337+0000","last_clean":"2023-12-16T21:57:45.264337+0000","last_became_active":"2023-12-16T21:57:43.509720+0000","last_became_peered":"2023-12-16T21:57:43.509720+0000","last_unstale":"2023-12-16T21:57:45.264337+0000","last_undegraded":"2023-12-16T21:57:45.264337+0000","last_fullsized":"2023-12-16T21:57:45.264337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:15.360923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252634+0000","last_change":"2023-12-16T21:57:42.518031+0000","last_active":"2023-12-16T21:57:45.252634+0000","last_peered":"2023-12-16T21:57:45.252634+0000","last_clean":"2023-12-16T21:57:45.252634+0000","last_became_active":"2023-12-16T21:57:42.517664+0000","last_became_peered":"2023-12-16T21:57:42.517664+0000","last_unstale":"2023-12-16T21:57:45.252634+0000","last_undegraded":"2023-12-16T21:57:45.252634+0000","last_fullsized":"2023-12-16T21:57:45.252634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252215+0000","last_change":"2023-12-16T21:57:43.493896+0000","last_active":"2023-12-16T21:57:45.252215+0000","last_peered":"2023-12-16T21:57:45.252215+0000","last_clean":"2023-12-16T21:57:45.252215+0000","last_became_active":"2023-12-16T21:57:43.492917+0000","last_became_peered":"2023-12-16T21:57:43.492917+0000","last_unstale":"2023-12-16T21:57:45.252215+0000","last_undegraded":"2023-12-16T21:57:45.252215+0000","last_fullsized":"2023-12-16T21:57:45.252215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:22.190906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263439+0000","last_change":"2023-12-16T21:57:42.517161+0000","last_active":"2023-12-16T21:57:45.263439+0000","last_peered":"2023-12-16T21:57:45.263439+0000","last_clean":"2023-12-16T21:57:45.263439+0000","last_became_active":"2023-12-16T21:57:42.516388+0000","last_became_peered":"2023-12-16T21:57:42.516388+0000","last_unstale":"2023-12-16T21:57:45.263439+0000","last_undegraded":"2023-12-16T21:57:45.263439+0000","last_fullsized":"2023-12-16T21:57:45.263439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:15.423681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264304+0000","last_change":"2023-12-16T21:57:43.512923+0000","last_active":"2023-12-16T21:57:45.264304+0000","last_peered":"2023-12-16T21:57:45.264304+0000","last_clean":"2023-12-16T21:57:45.264304+0000","last_became_active":"2023-12-16T21:57:43.512254+0000","last_became_peered":"2023-12-16T21:57:43.512254+0000","last_unstale":"2023-12-16T21:57:45.264304+0000","last_undegraded":"2023-12-16T21:57:45.264304+0000","last_fullsized":"2023-12-16T21:57:45.264304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:20.834561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457916+0000","last_change":"2023-12-16T21:57:42.511480+0000","last_active":"2023-12-16T21:57:44.457916+0000","last_peered":"2023-12-16T21:57:44.457916+0000","last_clean":"2023-12-16T21:57:44.457916+0000","last_became_active":"2023-12-16T21:57:42.510772+0000","last_became_peered":"2023-12-16T21:57:42.510772+0000","last_unstale":"2023-12-16T21:57:44.457916+0000","last_undegraded":"2023-12-16T21:57:44.457916+0000","last_fullsized":"2023-12-16T21:57:44.457916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457878+0000","last_change":"2023-12-16T21:57:43.493903+0000","last_active":"2023-12-16T21:57:44.457878+0000","last_peered":"2023-12-16T21:57:44.457878+0000","last_clean":"2023-12-16T21:57:44.457878+0000","last_became_active":"2023-12-16T21:57:43.493169+0000","last_became_peered":"2023-12-16T21:57:43.493169+0000","last_unstale":"2023-12-16T21:57:44.457878+0000","last_undegraded":"2023-12-16T21:57:44.457878+0000","last_fullsized":"2023-12-16T21:57:44.457878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.538705+0000","last_change":"2023-12-16T21:57:42.498884+0000","last_active":"2023-12-16T21:57:43.538705+0000","last_peered":"2023-12-16T21:57:43.538705+0000","last_clean":"2023-12-16T21:57:43.538705+0000","last_became_active":"2023-12-16T21:57:42.498452+0000","last_became_peered":"2023-12-16T21:57:42.498452+0000","last_unstale":"2023-12-16T21:57:43.538705+0000","last_undegraded":"2023-12-16T21:57:43.538705+0000","last_fullsized":"2023-12-16T21:57:43.538705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:37:03.298907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263857+0000","last_change":"2023-12-16T21:57:43.513792+0000","last_active":"2023-12-16T21:57:45.263857+0000","last_peered":"2023-12-16T21:57:45.263857+0000","last_clean":"2023-12-16T21:57:45.263857+0000","last_became_active":"2023-12-16T21:57:43.513001+0000","last_became_peered":"2023-12-16T21:57:43.513001+0000","last_unstale":"2023-12-16T21:57:45.263857+0000","last_undegraded":"2023-12-16T21:57:45.263857+0000","last_fullsized":"2023-12-16T21:57:45.263857+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459156+0000","last_change":"2023-12-16T21:57:42.512676+0000","last_active":"2023-12-16T21:57:44.459156+0000","last_peered":"2023-12-16T21:57:44.459156+0000","last_clean":"2023-12-16T21:57:44.459156+0000","last_became_active":"2023-12-16T21:57:42.512267+0000","last_became_peered":"2023-12-16T21:57:42.512267+0000","last_unstale":"2023-12-16T21:57:44.459156+0000","last_undegraded":"2023-12-16T21:57:44.459156+0000","last_fullsized":"2023-12-16T21:57:44.459156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:23:06.394772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264897+0000","last_change":"2023-12-16T21:57:43.517003+0000","last_active":"2023-12-16T21:57:45.264897+0000","last_peered":"2023-12-16T21:57:45.264897+0000","last_clean":"2023-12-16T21:57:45.264897+0000","last_became_active":"2023-12-16T21:57:43.516197+0000","last_became_peered":"2023-12-16T21:57:43.516197+0000","last_unstale":"2023-12-16T21:57:45.264897+0000","last_undegraded":"2023-12-16T21:57:45.264897+0000","last_fullsized":"2023-12-16T21:57:45.264897+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:48:39.071922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.534245+0000","last_change":"2023-12-16T21:57:42.512911+0000","last_active":"2023-12-16T21:57:43.534245+0000","last_peered":"2023-12-16T21:57:43.534245+0000","last_clean":"2023-12-16T21:57:43.534245+0000","last_became_active":"2023-12-16T21:57:42.512410+0000","last_became_peered":"2023-12-16T21:57:42.512410+0000","last_unstale":"2023-12-16T21:57:43.534245+0000","last_undegraded":"2023-12-16T21:57:43.534245+0000","last_fullsized":"2023-12-16T21:57:43.534245+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:00:58.952738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264838+0000","last_change":"2023-12-16T21:57:43.518693+0000","last_active":"2023-12-16T21:57:45.264838+0000","last_peered":"2023-12-16T21:57:45.264838+0000","last_clean":"2023-12-16T21:57:45.264838+0000","last_became_active":"2023-12-16T21:57:43.518184+0000","last_became_peered":"2023-12-16T21:57:43.518184+0000","last_unstale":"2023-12-16T21:57:45.264838+0000","last_undegraded":"2023-12-16T21:57:45.264838+0000","last_fullsized":"2023-12-16T21:57:45.264838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:47:38.932092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869962+0000","last_change":"2023-12-16T21:57:42.505084+0000","last_active":"2023-12-16T21:57:44.869962+0000","last_peered":"2023-12-16T21:57:44.869962+0000","last_clean":"2023-12-16T21:57:44.869962+0000","last_became_active":"2023-12-16T21:57:42.502948+0000","last_became_peered":"2023-12-16T21:57:42.502948+0000","last_unstale":"2023-12-16T21:57:44.869962+0000","last_undegraded":"2023-12-16T21:57:44.869962+0000","last_fullsized":"2023-12-16T21:57:44.869962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262484+0000","last_change":"2023-12-16T21:57:43.499808+0000","last_active":"2023-12-16T21:57:45.262484+0000","last_peered":"2023-12-16T21:57:45.262484+0000","last_clean":"2023-12-16T21:57:45.262484+0000","last_became_active":"2023-12-16T21:57:43.498922+0000","last_became_peered":"2023-12-16T21:57:43.498922+0000","last_unstale":"2023-12-16T21:57:45.262484+0000","last_undegraded":"2023-12-16T21:57:45.262484+0000","last_fullsized":"2023-12-16T21:57:45.262484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:04:09.033093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.465095+0000","last_change":"2023-12-16T21:57:42.509838+0000","last_active":"2023-12-16T21:57:43.465095+0000","last_peered":"2023-12-16T21:57:43.465095+0000","last_clean":"2023-12-16T21:57:43.465095+0000","last_became_active":"2023-12-16T21:57:42.508994+0000","last_became_peered":"2023-12-16T21:57:42.508994+0000","last_unstale":"2023-12-16T21:57:43.465095+0000","last_undegraded":"2023-12-16T21:57:43.465095+0000","last_fullsized":"2023-12-16T21:57:43.465095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:03:03.258824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264723+0000","last_change":"2023-12-16T21:57:43.516388+0000","last_active":"2023-12-16T21:57:45.264723+0000","last_peered":"2023-12-16T21:57:45.264723+0000","last_clean":"2023-12-16T21:57:45.264723+0000","last_became_active":"2023-12-16T21:57:43.515749+0000","last_became_peered":"2023-12-16T21:57:43.515749+0000","last_unstale":"2023-12-16T21:57:45.264723+0000","last_undegraded":"2023-12-16T21:57:45.264723+0000","last_fullsized":"2023-12-16T21:57:45.264723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:29.175289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.474023+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:57:43.474023+0000","last_peered":"2023-12-16T21:57:43.474023+0000","last_clean":"2023-12-16T21:57:43.474023+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:57:43.474023+0000","last_undegraded":"2023-12-16T21:57:43.474023+0000","last_fullsized":"2023-12-16T21:57:43.474023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261664+0000","last_change":"2023-12-16T21:57:42.508124+0000","last_active":"2023-12-16T21:57:45.261664+0000","last_peered":"2023-12-16T21:57:45.261664+0000","last_clean":"2023-12-16T21:57:45.261664+0000","last_became_active":"2023-12-16T21:57:42.507727+0000","last_became_peered":"2023-12-16T21:57:42.507727+0000","last_unstale":"2023-12-16T21:57:45.261664+0000","last_undegraded":"2023-12-16T21:57:45.261664+0000","last_fullsized":"2023-12-16T21:57:45.261664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262915+0000","last_change":"2023-12-16T21:57:43.490787+0000","last_active":"2023-12-16T21:57:45.262915+0000","last_peered":"2023-12-16T21:57:45.262915+0000","last_clean":"2023-12-16T21:57:45.262915+0000","last_became_active":"2023-12-16T21:57:43.490013+0000","last_became_peered":"2023-12-16T21:57:43.490013+0000","last_unstale":"2023-12-16T21:57:45.262915+0000","last_undegraded":"2023-12-16T21:57:45.262915+0000","last_fullsized":"2023-12-16T21:57:45.262915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262013+0000","last_change":"2023-12-16T21:57:42.523476+0000","last_active":"2023-12-16T21:57:45.262013+0000","last_peered":"2023-12-16T21:57:45.262013+0000","last_clean":"2023-12-16T21:57:45.262013+0000","last_became_active":"2023-12-16T21:57:42.522441+0000","last_became_peered":"2023-12-16T21:57:42.522441+0000","last_unstale":"2023-12-16T21:57:45.262013+0000","last_undegraded":"2023-12-16T21:57:45.262013+0000","last_fullsized":"2023-12-16T21:57:45.262013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:15.920999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252089+0000","last_change":"2023-12-16T21:57:43.508913+0000","last_active":"2023-12-16T21:57:45.252089+0000","last_peered":"2023-12-16T21:57:45.252089+0000","last_clean":"2023-12-16T21:57:45.252089+0000","last_became_active":"2023-12-16T21:57:43.508133+0000","last_became_peered":"2023-12-16T21:57:43.508133+0000","last_unstale":"2023-12-16T21:57:45.252089+0000","last_undegraded":"2023-12-16T21:57:45.252089+0000","last_fullsized":"2023-12-16T21:57:45.252089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:58.171558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.462961+0000","last_change":"2023-12-16T21:57:42.513117+0000","last_active":"2023-12-16T21:57:43.462961+0000","last_peered":"2023-12-16T21:57:43.462961+0000","last_clean":"2023-12-16T21:57:43.462961+0000","last_became_active":"2023-12-16T21:57:42.512449+0000","last_became_peered":"2023-12-16T21:57:42.512449+0000","last_unstale":"2023-12-16T21:57:43.462961+0000","last_undegraded":"2023-12-16T21:57:43.462961+0000","last_fullsized":"2023-12-16T21:57:43.462961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:55:25.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869573+0000","last_change":"2023-12-16T21:57:43.486971+0000","last_active":"2023-12-16T21:57:44.869573+0000","last_peered":"2023-12-16T21:57:44.869573+0000","last_clean":"2023-12-16T21:57:44.869573+0000","last_became_active":"2023-12-16T21:57:43.483854+0000","last_became_peered":"2023-12-16T21:57:43.483854+0000","last_unstale":"2023-12-16T21:57:44.869573+0000","last_undegraded":"2023-12-16T21:57:44.869573+0000","last_fullsized":"2023-12-16T21:57:44.869573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263608+0000","last_change":"2023-12-16T21:57:42.501969+0000","last_active":"2023-12-16T21:57:45.263608+0000","last_peered":"2023-12-16T21:57:45.263608+0000","last_clean":"2023-12-16T21:57:45.263608+0000","last_became_active":"2023-12-16T21:57:42.501059+0000","last_became_peered":"2023-12-16T21:57:42.501059+0000","last_unstale":"2023-12-16T21:57:45.263608+0000","last_undegraded":"2023-12-16T21:57:45.263608+0000","last_fullsized":"2023-12-16T21:57:45.263608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:09.342194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459101+0000","last_change":"2023-12-16T21:57:43.506507+0000","last_active":"2023-12-16T21:57:44.459101+0000","last_peered":"2023-12-16T21:57:44.459101+0000","last_clean":"2023-12-16T21:57:44.459101+0000","last_became_active":"2023-12-16T21:57:43.505685+0000","last_became_peered":"2023-12-16T21:57:43.505685+0000","last_unstale":"2023-12-16T21:57:44.459101+0000","last_undegraded":"2023-12-16T21:57:44.459101+0000","last_fullsized":"2023-12-16T21:57:44.459101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:40.013144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869952+0000","last_change":"2023-12-16T21:57:42.503616+0000","last_active":"2023-12-16T21:57:44.869952+0000","last_peered":"2023-12-16T21:57:44.869952+0000","last_clean":"2023-12-16T21:57:44.869952+0000","last_became_active":"2023-12-16T21:57:42.502114+0000","last_became_peered":"2023-12-16T21:57:42.502114+0000","last_unstale":"2023-12-16T21:57:44.869952+0000","last_undegraded":"2023-12-16T21:57:44.869952+0000","last_fullsized":"2023-12-16T21:57:44.869952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:57.160385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262539+0000","last_change":"2023-12-16T21:57:43.500130+0000","last_active":"2023-12-16T21:57:45.262539+0000","last_peered":"2023-12-16T21:57:45.262539+0000","last_clean":"2023-12-16T21:57:45.262539+0000","last_became_active":"2023-12-16T21:57:43.499128+0000","last_became_peered":"2023-12-16T21:57:43.499128+0000","last_unstale":"2023-12-16T21:57:45.262539+0000","last_undegraded":"2023-12-16T21:57:45.262539+0000","last_fullsized":"2023-12-16T21:57:45.262539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:20.834561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.265035+0000","last_change":"2023-12-16T21:57:42.526337+0000","last_active":"2023-12-16T21:57:45.265035+0000","last_peered":"2023-12-16T21:57:45.265035+0000","last_clean":"2023-12-16T21:57:45.265035+0000","last_became_active":"2023-12-16T21:57:42.525796+0000","last_became_peered":"2023-12-16T21:57:42.525796+0000","last_unstale":"2023-12-16T21:57:45.265035+0000","last_undegraded":"2023-12-16T21:57:45.265035+0000","last_fullsized":"2023-12-16T21:57:45.265035+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:00:29.774694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870185+0000","last_change":"2023-12-16T21:57:43.477509+0000","last_active":"2023-12-16T21:57:44.870185+0000","last_peered":"2023-12-16T21:57:44.870185+0000","last_clean":"2023-12-16T21:57:44.870185+0000","last_became_active":"2023-12-16T21:57:43.476854+0000","last_became_peered":"2023-12-16T21:57:43.476854+0000","last_unstale":"2023-12-16T21:57:44.870185+0000","last_undegraded":"2023-12-16T21:57:44.870185+0000","last_fullsized":"2023-12-16T21:57:44.870185+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870093+0000","last_change":"2023-12-16T21:57:42.499802+0000","last_active":"2023-12-16T21:57:44.870093+0000","last_peered":"2023-12-16T21:57:44.870093+0000","last_clean":"2023-12-16T21:57:44.870093+0000","last_became_active":"2023-12-16T21:57:42.498817+0000","last_became_peered":"2023-12-16T21:57:42.498817+0000","last_unstale":"2023-12-16T21:57:44.870093+0000","last_undegraded":"2023-12-16T21:57:44.870093+0000","last_fullsized":"2023-12-16T21:57:44.870093+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263239+0000","last_change":"2023-12-16T21:57:43.490695+0000","last_active":"2023-12-16T21:57:45.263239+0000","last_peered":"2023-12-16T21:57:45.263239+0000","last_clean":"2023-12-16T21:57:45.263239+0000","last_became_active":"2023-12-16T21:57:43.489927+0000","last_became_peered":"2023-12-16T21:57:43.489927+0000","last_unstale":"2023-12-16T21:57:45.263239+0000","last_undegraded":"2023-12-16T21:57:45.263239+0000","last_fullsized":"2023-12-16T21:57:45.263239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458559+0000","last_change":"2023-12-16T21:57:42.523142+0000","last_active":"2023-12-16T21:57:44.458559+0000","last_peered":"2023-12-16T21:57:44.458559+0000","last_clean":"2023-12-16T21:57:44.458559+0000","last_became_active":"2023-12-16T21:57:42.521597+0000","last_became_peered":"2023-12-16T21:57:42.521597+0000","last_unstale":"2023-12-16T21:57:44.458559+0000","last_undegraded":"2023-12-16T21:57:44.458559+0000","last_fullsized":"2023-12-16T21:57:44.458559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252927+0000","last_change":"2023-12-16T21:57:43.483454+0000","last_active":"2023-12-16T21:57:45.252927+0000","last_peered":"2023-12-16T21:57:45.252927+0000","last_clean":"2023-12-16T21:57:45.252927+0000","last_became_active":"2023-12-16T21:57:43.482451+0000","last_became_peered":"2023-12-16T21:57:43.482451+0000","last_unstale":"2023-12-16T21:57:45.252927+0000","last_undegraded":"2023-12-16T21:57:45.252927+0000","last_fullsized":"2023-12-16T21:57:45.252927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261795+0000","last_change":"2023-12-16T21:57:42.522091+0000","last_active":"2023-12-16T21:57:45.261795+0000","last_peered":"2023-12-16T21:57:45.261795+0000","last_clean":"2023-12-16T21:57:45.261795+0000","last_became_active":"2023-12-16T21:57:42.521328+0000","last_became_peered":"2023-12-16T21:57:42.521328+0000","last_unstale":"2023-12-16T21:57:45.261795+0000","last_undegraded":"2023-12-16T21:57:45.261795+0000","last_fullsized":"2023-12-16T21:57:45.261795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457982+0000","last_change":"2023-12-16T21:57:43.494064+0000","last_active":"2023-12-16T21:57:44.457982+0000","last_peered":"2023-12-16T21:57:44.457982+0000","last_clean":"2023-12-16T21:57:44.457982+0000","last_became_active":"2023-12-16T21:57:43.493374+0000","last_became_peered":"2023-12-16T21:57:43.493374+0000","last_unstale":"2023-12-16T21:57:44.457982+0000","last_undegraded":"2023-12-16T21:57:44.457982+0000","last_fullsized":"2023-12-16T21:57:44.457982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:09.494346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869828+0000","last_change":"2023-12-16T21:57:42.508931+0000","last_active":"2023-12-16T21:57:44.869828+0000","last_peered":"2023-12-16T21:57:44.869828+0000","last_clean":"2023-12-16T21:57:44.869828+0000","last_became_active":"2023-12-16T21:57:42.508438+0000","last_became_peered":"2023-12-16T21:57:42.508438+0000","last_unstale":"2023-12-16T21:57:44.869828+0000","last_undegraded":"2023-12-16T21:57:44.869828+0000","last_fullsized":"2023-12-16T21:57:44.869828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:39.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262488+0000","last_change":"2023-12-16T21:57:43.485012+0000","last_active":"2023-12-16T21:57:45.262488+0000","last_peered":"2023-12-16T21:57:45.262488+0000","last_clean":"2023-12-16T21:57:45.262488+0000","last_became_active":"2023-12-16T21:57:43.484177+0000","last_became_peered":"2023-12-16T21:57:43.484177+0000","last_unstale":"2023-12-16T21:57:45.262488+0000","last_undegraded":"2023-12-16T21:57:45.262488+0000","last_fullsized":"2023-12-16T21:57:45.262488+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869858+0000","last_change":"2023-12-16T21:57:42.504338+0000","last_active":"2023-12-16T21:57:44.869858+0000","last_peered":"2023-12-16T21:57:44.869858+0000","last_clean":"2023-12-16T21:57:44.869858+0000","last_became_active":"2023-12-16T21:57:42.503125+0000","last_became_peered":"2023-12-16T21:57:42.503125+0000","last_unstale":"2023-12-16T21:57:44.869858+0000","last_undegraded":"2023-12-16T21:57:44.869858+0000","last_fullsized":"2023-12-16T21:57:44.869858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:23:06.394772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262761+0000","last_change":"2023-12-16T21:57:43.499585+0000","last_active":"2023-12-16T21:57:45.262761+0000","last_peered":"2023-12-16T21:57:45.262761+0000","last_clean":"2023-12-16T21:57:45.262761+0000","last_became_active":"2023-12-16T21:57:43.498790+0000","last_became_peered":"2023-12-16T21:57:43.498790+0000","last_unstale":"2023-12-16T21:57:45.262761+0000","last_undegraded":"2023-12-16T21:57:45.262761+0000","last_fullsized":"2023-12-16T21:57:45.262761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264439+0000","last_change":"2023-12-16T21:57:42.522249+0000","last_active":"2023-12-16T21:57:45.264439+0000","last_peered":"2023-12-16T21:57:45.264439+0000","last_clean":"2023-12-16T21:57:45.264439+0000","last_became_active":"2023-12-16T21:57:42.521585+0000","last_became_peered":"2023-12-16T21:57:42.521585+0000","last_unstale":"2023-12-16T21:57:45.264439+0000","last_undegraded":"2023-12-16T21:57:45.264439+0000","last_fullsized":"2023-12-16T21:57:45.264439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:22.825415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457737+0000","last_change":"2023-12-16T21:57:43.495929+0000","last_active":"2023-12-16T21:57:44.457737+0000","last_peered":"2023-12-16T21:57:44.457737+0000","last_clean":"2023-12-16T21:57:44.457737+0000","last_became_active":"2023-12-16T21:57:43.494696+0000","last_became_peered":"2023-12-16T21:57:43.494696+0000","last_unstale":"2023-12-16T21:57:44.457737+0000","last_undegraded":"2023-12-16T21:57:44.457737+0000","last_fullsized":"2023-12-16T21:57:44.457737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262280+0000","last_change":"2023-12-16T21:57:43.496545+0000","last_active":"2023-12-16T21:57:45.262280+0000","last_peered":"2023-12-16T21:57:45.262280+0000","last_clean":"2023-12-16T21:57:45.262280+0000","last_became_active":"2023-12-16T21:57:43.495846+0000","last_became_peered":"2023-12-16T21:57:43.495846+0000","last_unstale":"2023-12-16T21:57:45.262280+0000","last_undegraded":"2023-12-16T21:57:45.262280+0000","last_fullsized":"2023-12-16T21:57:45.262280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264212+0000","last_change":"2023-12-16T21:57:42.524077+0000","last_active":"2023-12-16T21:57:45.264212+0000","last_peered":"2023-12-16T21:57:45.264212+0000","last_clean":"2023-12-16T21:57:45.264212+0000","last_became_active":"2023-12-16T21:57:42.523590+0000","last_became_peered":"2023-12-16T21:57:42.523590+0000","last_unstale":"2023-12-16T21:57:45.264212+0000","last_undegraded":"2023-12-16T21:57:45.264212+0000","last_fullsized":"2023-12-16T21:57:45.264212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:57.160385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262278+0000","last_change":"2023-12-16T21:57:43.508947+0000","last_active":"2023-12-16T21:57:45.262278+0000","last_peered":"2023-12-16T21:57:45.262278+0000","last_clean":"2023-12-16T21:57:45.262278+0000","last_became_active":"2023-12-16T21:57:43.508424+0000","last_became_peered":"2023-12-16T21:57:43.508424+0000","last_unstale":"2023-12-16T21:57:45.262278+0000","last_undegraded":"2023-12-16T21:57:45.262278+0000","last_fullsized":"2023-12-16T21:57:45.262278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264760+0000","last_change":"2023-12-16T21:57:42.522477+0000","last_active":"2023-12-16T21:57:45.264760+0000","last_peered":"2023-12-16T21:57:45.264760+0000","last_clean":"2023-12-16T21:57:45.264760+0000","last_became_active":"2023-12-16T21:57:42.521732+0000","last_became_peered":"2023-12-16T21:57:42.521732+0000","last_unstale":"2023-12-16T21:57:45.264760+0000","last_undegraded":"2023-12-16T21:57:45.264760+0000","last_fullsized":"2023-12-16T21:57:45.264760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252926+0000","last_change":"2023-12-16T21:57:43.481470+0000","last_active":"2023-12-16T21:57:45.252926+0000","last_peered":"2023-12-16T21:57:45.252926+0000","last_clean":"2023-12-16T21:57:45.252926+0000","last_became_active":"2023-12-16T21:57:43.480541+0000","last_became_peered":"2023-12-16T21:57:43.480541+0000","last_unstale":"2023-12-16T21:57:45.252926+0000","last_undegraded":"2023-12-16T21:57:45.252926+0000","last_fullsized":"2023-12-16T21:57:45.252926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263338+0000","last_change":"2023-12-16T21:57:42.512776+0000","last_active":"2023-12-16T21:57:45.263338+0000","last_peered":"2023-12-16T21:57:45.263338+0000","last_clean":"2023-12-16T21:57:45.263338+0000","last_became_active":"2023-12-16T21:57:42.512393+0000","last_became_peered":"2023-12-16T21:57:42.512393+0000","last_unstale":"2023-12-16T21:57:45.263338+0000","last_undegraded":"2023-12-16T21:57:45.263338+0000","last_fullsized":"2023-12-16T21:57:45.263338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263375+0000","last_change":"2023-12-16T21:57:43.499571+0000","last_active":"2023-12-16T21:57:45.263375+0000","last_peered":"2023-12-16T21:57:45.263375+0000","last_clean":"2023-12-16T21:57:45.263375+0000","last_became_active":"2023-12-16T21:57:43.498880+0000","last_became_peered":"2023-12-16T21:57:43.498880+0000","last_unstale":"2023-12-16T21:57:45.263375+0000","last_undegraded":"2023-12-16T21:57:45.263375+0000","last_fullsized":"2023-12-16T21:57:45.263375+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.253109+0000","last_change":"2023-12-16T21:57:42.522619+0000","last_active":"2023-12-16T21:57:45.253109+0000","last_peered":"2023-12-16T21:57:45.253109+0000","last_clean":"2023-12-16T21:57:45.253109+0000","last_became_active":"2023-12-16T21:57:42.522259+0000","last_became_peered":"2023-12-16T21:57:42.522259+0000","last_unstale":"2023-12-16T21:57:45.253109+0000","last_undegraded":"2023-12-16T21:57:45.253109+0000","last_fullsized":"2023-12-16T21:57:45.253109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476742,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":827392,"data_stored":670753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.127}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2290000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]}]},{"osd":6,"up_from":16,"seq":68719476741,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.974}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1820000000000004}]}]},{"osd":5,"up_from":16,"seq":68719476742,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":79893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.214}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]}]},{"osd":4,"up_from":16,"seq":68719476740,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":79929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]}]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":75115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.226}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":237568,"data_stored":79963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":823296,"data_stored":664509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1559999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":73503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0720000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:57:47.953 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-16T21:57:47.953 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-16T21:57:47.953 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-16T21:57:47.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-16T21:57:47.953 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-16T21:57:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6b5de640 1 Processor -- start 2023-12-16T21:57:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6b5de640 1 -- start start 2023-12-16T21:57:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6b5de640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640b35b0 0x7f1c640d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6b5de640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1c640cfe90 0x7f1c6415f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6b5de640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640d3880 con 0x7f1c640b3c70 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6b5de640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c640d4090 con 0x7f1c640cfe90 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6b5de640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640cdc80 con 0x7f1c640b35b0 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6937a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640b35b0 0x7f1c640d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c69b7b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1c640cfe90 0x7f1c6415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c68b79640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1c640b3c70 0x7f1c640cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:58770/0 (socket says 172.21.15.28:58770) 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c69b7b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1c640cfe90 0x7f1c6415f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:37620/0 (socket says 172.21.15.28:37620) 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c68b79640 1 -- 172.21.15.28:0/3382095096 learned_addr learned my addr 172.21.15.28:0/3382095096 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6937a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640b35b0 0x7f1c640d0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:59830/0 (socket says 172.21.15.28:59830) 2023-12-16T21:57:48.122 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c69b7b640 1 -- 172.21.15.28:0/3382095096 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640b35b0 msgr2=0x7f1c640d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:48.122 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c69b7b640 1 --2- 172.21.15.28:0/3382095096 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640b35b0 0x7f1c640d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.122 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c69b7b640 1 -- 172.21.15.28:0/3382095096 >> v1:172.21.15.122:6789/0 conn(0x7f1c640b3c70 legacy=0x7f1c640cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.122 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c69b7b640 1 -- 172.21.15.28:0/3382095096 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640d3ad0 con 0x7f1c640cfe90 2023-12-16T21:57:48.122 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.120+0000 7f1c6937a640 1 --2- 172.21.15.28:0/3382095096 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640b35b0 0x7f1c640d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:48.123 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c69b7b640 1 --2- 172.21.15.28:0/3382095096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640cfe90 0x7f1c6415f060 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009d90 tx=0x7f1c5c01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edb4643b0572f004 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.123 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c5bfff640 1 -- 172.21.15.28:0/3382095096 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0a5a10 con 0x7f1c640cfe90 2023-12-16T21:57:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c5bfff640 1 -- 172.21.15.28:0/3382095096 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c0a5b70 con 0x7f1c640cfe90 2023-12-16T21:57:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c5bfff640 1 -- 172.21.15.28:0/3382095096 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0ad440 con 0x7f1c640cfe90 2023-12-16T21:57:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3382095096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640cfe90 msgr2=0x7f1c6415f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3382095096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640cfe90 0x7f1c6415f060 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009d90 tx=0x7f1c5c01c950 comp rx=0 tx=0).stop 2023-12-16T21:57:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3382095096 shutdown_connections 2023-12-16T21:57:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3382095096 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640cfe90 0x7f1c6415f060 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3382095096 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640b35b0 0x7f1c640d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3382095096 >> 172.21.15.28:0/3382095096 conn(0x7f1c6409c4b0 msgr2=0x7f1c640d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3382095096 shutdown_connections 2023-12-16T21:57:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3382095096 wait complete. 2023-12-16T21:57:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 Processor -- start 2023-12-16T21:57:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.124+0000 7f1c6b5de640 1 -- start start 2023-12-16T21:57:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c640b35b0 0x7f1c640b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640b3c70 0x7f1c640b80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640cfe90 0x7f1c640b8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640ccd80 con 0x7f1c640b35b0 2023-12-16T21:57:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640cd1a0 con 0x7f1c640b3c70 2023-12-16T21:57:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640b5f20 con 0x7f1c640cfe90 2023-12-16T21:57:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c640b35b0 0x7f1c640b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c640b35b0 0x7f1c640b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45858/0 (socket says 172.21.15.28:45858) 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 -- 172.21.15.28:0/3045313452 learned_addr learned my addr 172.21.15.28:0/3045313452 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c68b79640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640b3c70 0x7f1c640b80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c69b7b640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640cfe90 0x7f1c640b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 -- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640cfe90 msgr2=0x7f1c640b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640cfe90 0x7f1c640b8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 -- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640b3c70 msgr2=0x7f1c640b80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640b3c70 0x7f1c640b80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 -- 172.21.15.28:0/3045313452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c60009420 con 0x7f1c640b35b0 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6937a640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c640b35b0 0x7f1c640b5840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000e9b0 tx=0x7f1c6000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c59ffb640 1 -- 172.21.15.28:0/3045313452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6000ca00 con 0x7f1c640b35b0 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c59ffb640 1 -- 172.21.15.28:0/3045313452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6000cb60 con 0x7f1c640b35b0 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5c009a70 con 0x7f1c640b35b0 2023-12-16T21:57:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c59ffb640 1 -- 172.21.15.28:0/3045313452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60013470 con 0x7f1c640b35b0 2023-12-16T21:57:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c640b7530 con 0x7f1c640b35b0 2023-12-16T21:57:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c640ccff0 con 0x7f1c640b35b0 2023-12-16T21:57:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c59ffb640 1 -- 172.21.15.28:0/3045313452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1c600135d0 con 0x7f1c640b35b0 2023-12-16T21:57:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c59ffb640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1c480664a0 0x7f1c48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c59ffb640 1 -- 172.21.15.28:0/3045313452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c60084030 con 0x7f1c640b35b0 2023-12-16T21:57:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.128+0000 7f1c68b79640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1c480664a0 0x7f1c48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.132+0000 7f1c68b79640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1c480664a0 0x7f1c48068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c640b57d0 tx=0x7f1c54009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.136+0000 7f1c59ffb640 1 -- 172.21.15.28:0/3045313452 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f1c60051540 con 0x7f1c640b35b0 2023-12-16T21:57:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.332+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f1c64069440 con 0x7f1c640b35b0 2023-12-16T21:57:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.336+0000 7f1c59ffb640 1 -- 172.21.15.28:0/3045313452 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164438 (secure 0 0 0) 0x7f1c60054b60 con 0x7f1c640b35b0 2023-12-16T21:57:48.339 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:48.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1c480664a0 msgr2=0x7f1c48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1c480664a0 0x7f1c48068960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c640b57d0 tx=0x7f1c54009040 comp rx=0 tx=0).stop 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c640b35b0 msgr2=0x7f1c640b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c640b35b0 0x7f1c640b5840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1c6000e9b0 tx=0x7f1c6000ee70 comp rx=0 tx=0).stop 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 shutdown_connections 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1c480664a0 0x7f1c48068960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1c640cfe90 0x7f1c640b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c640b3c70 0x7f1c640b80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 --2- 172.21.15.28:0/3045313452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c640b35b0 0x7f1c640b5840 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 >> 172.21.15.28:0/3045313452 conn(0x7f1c6409c4b0 msgr2=0x7f1c640b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 shutdown_connections 2023-12-16T21:57:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.340+0000 7f1c6b5de640 1 -- 172.21.15.28:0/3045313452 wait complete. 2023-12-16T21:57:48.360 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":3228142796},{"type":"v1","addr":"172.21.15.122:6833","nonce":3228142796}]},"active_addr":"172.21.15.122:6833/3228142796","active_change":"2023-12-16T21:57:23.368194+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4098,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":3906357667}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":924131468}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":643677421}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.122:0","nonce":190066129}]}]} 2023-12-16T21:57:48.363 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-16T21:57:48.363 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-16T21:57:48.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28e0228640 1 Processor -- start 2023-12-16T21:57:48.493 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28e0228640 1 -- start start 2023-12-16T21:57:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28e0228640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 0x7f28d80d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28e0228640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28d80b3c70 0x7f28d815f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28e0228640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80d3880 con 0x7f28d80b35b0 2023-12-16T21:57:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28e0228640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d80d4090 con 0x7f28d80b3c70 2023-12-16T21:57:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28e0228640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80cdc80 con 0x7f28d80cfe90 2023-12-16T21:57:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28ddfc4640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f28d80b35b0 0x7f28d80cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51138/0 (socket says 172.21.15.28:51138) 2023-12-16T21:57:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28ddfc4640 1 -- 172.21.15.28:0/1703715961 learned_addr learned my addr 172.21.15.28:0/1703715961 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28de7c5640 1 --2- 172.21.15.28:0/1703715961 >> v2:172.21.15.28:3300/0 conn(0x7f28d80b3c70 0x7f28d815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28dd7c3640 1 --2- 172.21.15.28:0/1703715961 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 0x7f28d80d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28de7c5640 1 -- 172.21.15.28:0/1703715961 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 msgr2=0x7f28d80d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28de7c5640 1 --2- 172.21.15.28:0/1703715961 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 0x7f28d80d0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28de7c5640 1 -- 172.21.15.28:0/1703715961 >> v1:172.21.15.28:6789/0 conn(0x7f28d80b35b0 legacy=0x7f28d80cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28dcfc2640 1 -- 172.21.15.28:0/1703715961 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075595862 0 0) 0x7f28d80d3880 con 0x7f28d80b35b0 2023-12-16T21:57:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.492+0000 7f28de7c5640 1 -- 172.21.15.28:0/1703715961 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d80d3ad0 con 0x7f28d80b3c70 2023-12-16T21:57:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28de7c5640 1 --2- 172.21.15.28:0/1703715961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b3c70 0x7f28d815f060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009d90 tx=0x7f28d001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cacea3f45da93ec server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28dcfc2640 1 -- 172.21.15.28:0/1703715961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00a5980 con 0x7f28d80b3c70 2023-12-16T21:57:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28dcfc2640 1 -- 172.21.15.28:0/1703715961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d00a5ae0 con 0x7f28d80b3c70 2023-12-16T21:57:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28dcfc2640 1 -- 172.21.15.28:0/1703715961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00a5e70 con 0x7f28d80b3c70 2023-12-16T21:57:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 -- 172.21.15.28:0/1703715961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b3c70 msgr2=0x7f28d815f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 --2- 172.21.15.28:0/1703715961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b3c70 0x7f28d815f060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009d90 tx=0x7f28d001c8c0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 -- 172.21.15.28:0/1703715961 shutdown_connections 2023-12-16T21:57:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 --2- 172.21.15.28:0/1703715961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b3c70 0x7f28d815f060 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 --2- 172.21.15.28:0/1703715961 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 0x7f28d80d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 -- 172.21.15.28:0/1703715961 >> 172.21.15.28:0/1703715961 conn(0x7f28d809c4b0 msgr2=0x7f28d80d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 -- 172.21.15.28:0/1703715961 shutdown_connections 2023-12-16T21:57:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 -- 172.21.15.28:0/1703715961 wait complete. 2023-12-16T21:57:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 Processor -- start 2023-12-16T21:57:48.499 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.496+0000 7f28e0228640 1 -- start start 2023-12-16T21:57:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b35b0 0x7f28d815b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28d80b3c70 0x7f28d80cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 0x7f28d80a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d815b0b0 con 0x7f28d80b35b0 2023-12-16T21:57:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80cc720 con 0x7f28d80b3c70 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80cd1c0 con 0x7f28d80cfe90 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b35b0 0x7f28d815b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b35b0 0x7f28d815b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45872/0 (socket says 172.21.15.28:45872) 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 -- 172.21.15.28:0/2804926816 learned_addr learned my addr 172.21.15.28:0/2804926816 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28de7c5640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 0x7f28d80a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28dd7c3640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28d80b3c70 0x7f28d80cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 -- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 msgr2=0x7f28d80a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 0x7f28d80a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 -- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28d80b3c70 msgr2=0x7f28d80cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28d80b3c70 0x7f28d80cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 -- 172.21.15.28:0/2804926816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d0009a70 con 0x7f28d80b35b0 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ddfc4640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b35b0 0x7f28d815b230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28d400c480 tx=0x7f28d400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ca7fc640 1 -- 172.21.15.28:0/2804926816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d40074e0 con 0x7f28d80b35b0 2023-12-16T21:57:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ca7fc640 1 -- 172.21.15.28:0/2804926816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d4007660 con 0x7f28d80b35b0 2023-12-16T21:57:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ca7fc640 1 -- 172.21.15.28:0/2804926816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4004a30 con 0x7f28d80b35b0 2023-12-16T21:57:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d80cc920 con 0x7f28d80b35b0 2023-12-16T21:57:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d80cccb0 con 0x7f28d80b35b0 2023-12-16T21:57:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ca7fc640 1 -- 172.21.15.28:0/2804926816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f28d4004b90 con 0x7f28d80b35b0 2023-12-16T21:57:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d8069440 con 0x7f28d80b35b0 2023-12-16T21:57:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ca7fc640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28bc066470 0x7f28bc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28ca7fc640 1 -- 172.21.15.28:0/2804926816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28d4082fa0 con 0x7f28d80b35b0 2023-12-16T21:57:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.500+0000 7f28dd7c3640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28bc066470 0x7f28bc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.504+0000 7f28dd7c3640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28bc066470 0x7f28bc068930 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f28d81681b0 tx=0x7f28cc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.508+0000 7f28ca7fc640 1 -- 172.21.15.28:0/2804926816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f28d40b4290 con 0x7f28d80b35b0 2023-12-16T21:57:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.660+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28d80d0720 con 0x7f28d80b35b0 2023-12-16T21:57:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28ca7fc640 1 -- 172.21.15.28:0/2804926816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f28d4053f40 con 0x7f28d80b35b0 2023-12-16T21:57:48.664 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:48.664 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28bc066470 msgr2=0x7f28bc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28bc066470 0x7f28bc068930 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f28d81681b0 tx=0x7f28cc008040 comp rx=0 tx=0).stop 2023-12-16T21:57:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b35b0 msgr2=0x7f28d815b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b35b0 0x7f28d815b230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28d400c480 tx=0x7f28d400c940 comp rx=0 tx=0).stop 2023-12-16T21:57:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 shutdown_connections 2023-12-16T21:57:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28bc066470 0x7f28bc068930 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f28d80cfe90 0x7f28d80a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28d80b3c70 0x7f28d80cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 --2- 172.21.15.28:0/2804926816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28d80b35b0 0x7f28d815b230 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 >> 172.21.15.28:0/2804926816 conn(0x7f28d809c4b0 msgr2=0x7f28d80b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 shutdown_connections 2023-12-16T21:57:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.664+0000 7f28e0228640 1 -- 172.21.15.28:0/2804926816 wait complete. 2023-12-16T21:57:48.682 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-16T21:57:48.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:48.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc3864640 1 Processor -- start 2023-12-16T21:57:48.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc3864640 1 -- start start 2023-12-16T21:57:48.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc3864640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0cfe90 0x7f4bbc0d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc3864640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0d3880 con 0x7f4bbc0b35b0 2023-12-16T21:57:48.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc3864640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0cdc80 con 0x7f4bbc0b3c70 2023-12-16T21:57:48.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc3864640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0d4090 con 0x7f4bbc0cfe90 2023-12-16T21:57:48.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc1600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4bbc0b35b0 0x7f4bbc0cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51148/0 (socket says 172.21.15.28:51148) 2023-12-16T21:57:48.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc1600640 1 -- 172.21.15.28:0/2340846401 learned_addr learned my addr 172.21.15.28:0/2340846401 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:48.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095899055 0 0) 0x7f4bbc0d3880 con 0x7f4bbc0b35b0 2023-12-16T21:57:48.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bc0dff640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0cfe90 0x7f4bbc0d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.792+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba8003530 con 0x7f4bbc0b35b0 2023-12-16T21:57:48.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3281677066 0 0) 0x7f4ba8003530 con 0x7f4bbc0b35b0 2023-12-16T21:57:48.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0cfe90 msgr2=0x7f4bbc0d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0cfe90 0x7f4bbc0d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 >> v1:172.21.15.122:6789/0 conn(0x7f4bbc0b3c70 legacy=0x7f4bbc0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0d3ad0 con 0x7f4bbc0b35b0 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f4bb8001a00 con 0x7f4bbc0b35b0 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 >> v1:172.21.15.28:6789/0 conn(0x7f4bbc0b35b0 legacy=0x7f4bbc0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4ba8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0cfe90 0x7f4ba80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba8004d10 0x7f4ba8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0d4090 con 0x7f4ba80039b0 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba800bb20 con 0x7f4ba8004d10 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba800bce0 con 0x7f4bbc0cfe90 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1600640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4ba8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1e01640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba8004d10 0x7f4ba8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1600640 1 -- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0cfe90 msgr2=0x7f4ba80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1600640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0cfe90 0x7f4ba80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1600640 1 -- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba8004d10 msgr2=0x7f4ba8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1600640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba8004d10 0x7f4ba8005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1600640 1 -- 172.21.15.28:0/2340846401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba8003530 con 0x7f4ba80039b0 2023-12-16T21:57:48.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1600640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4ba8003dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800a760 tx=0x7f4bb801d230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2027b0ad9ae343f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80ad630 con 0x7f4ba80039b0 2023-12-16T21:57:48.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb80a8070 con 0x7f4ba80039b0 2023-12-16T21:57:48.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bb3fff640 1 -- 172.21.15.28:0/2340846401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80ada10 con 0x7f4ba80039b0 2023-12-16T21:57:48.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc3864640 1 -- 172.21.15.28:0/2340846401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 msgr2=0x7f4ba8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc3864640 1 --2- 172.21.15.28:0/2340846401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4ba8003dd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800a760 tx=0x7f4bb801d230 comp rx=0 tx=0).stop 2023-12-16T21:57:48.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc1e01640 1 -- 172.21.15.28:0/2340846401 reap_dead start 2023-12-16T21:57:48.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc3864640 1 -- 172.21.15.28:0/2340846401 shutdown_connections 2023-12-16T21:57:48.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.796+0000 7f4bc3864640 1 -- 172.21.15.28:0/2340846401 >> 172.21.15.28:0/2340846401 conn(0x7f4bbc09c4b0 msgr2=0x7f4bbc0d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:48.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- 172.21.15.28:0/2340846401 shutdown_connections 2023-12-16T21:57:48.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- 172.21.15.28:0/2340846401 wait complete. 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 Processor -- start 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- start start 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4bbc0cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0ca240 0x7f4bbc156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bbc157200 0x7f4bbc0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc157030 con 0x7f4ba80039b0 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0ca790 con 0x7f4bbc157200 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0caa70 con 0x7f4bbc0ca240 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4bbc0cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4bbc0cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45902/0 (socket says 172.21.15.28:45902) 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 -- 172.21.15.28:0/151151607 learned_addr learned my addr 172.21.15.28:0/151151607 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1e01640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bbc157200 0x7f4bbc0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc0dff640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0ca240 0x7f4bbc156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 -- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0ca240 msgr2=0x7f4bbc156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0ca240 0x7f4bbc156af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 -- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bbc157200 msgr2=0x7f4bbc0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bbc157200 0x7f4bbc0cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 -- 172.21.15.28:0/151151607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0cad50 con 0x7f4ba80039b0 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc1600640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4bbc0cb830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800a760 tx=0x7f4ba8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bb1ffb640 1 -- 172.21.15.28:0/151151607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8001940 con 0x7f4ba80039b0 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bb1ffb640 1 -- 172.21.15.28:0/151151607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8004e30 con 0x7f4ba80039b0 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc0cb030 con 0x7f4ba80039b0 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bb1ffb640 1 -- 172.21.15.28:0/151151607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb80ad650 con 0x7f4ba80039b0 2023-12-16T21:57:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc157910 con 0x7f4ba80039b0 2023-12-16T21:57:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.800+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc069440 con 0x7f4ba80039b0 2023-12-16T21:57:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.804+0000 7f4bb1ffb640 1 -- 172.21.15.28:0/151151607 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4bb80aa030 con 0x7f4ba80039b0 2023-12-16T21:57:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.804+0000 7f4bb1ffb640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4ba0066470 0x7f4ba0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.804+0000 7f4bb1ffb640 1 -- 172.21.15.28:0/151151607 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4bb811e330 con 0x7f4ba80039b0 2023-12-16T21:57:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.804+0000 7f4bc0dff640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4ba0066470 0x7f4ba0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.804+0000 7f4bc0dff640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4ba0066470 0x7f4ba0068930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc1fa290 tx=0x7f4bac008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:48.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.808+0000 7f4bb1ffb640 1 -- 172.21.15.28:0/151151607 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4bb80eb7b0 con 0x7f4ba80039b0 2023-12-16T21:57:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.960+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bbc0d0720 con 0x7f4ba80039b0 2023-12-16T21:57:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.960+0000 7f4bb1ffb640 1 -- 172.21.15.28:0/151151607 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f4bb80eedd0 con 0x7f4ba80039b0 2023-12-16T21:57:48.962 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:48.962 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4ba0066470 msgr2=0x7f4ba0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4ba0066470 0x7f4ba0068930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc1fa290 tx=0x7f4bac008040 comp rx=0 tx=0).stop 2023-12-16T21:57:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 msgr2=0x7f4bbc0cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4bbc0cb830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4bb800a760 tx=0x7f4ba8005800 comp rx=0 tx=0).stop 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 shutdown_connections 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4ba0066470 0x7f4ba0068930 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4bbc157200 0x7f4bbc0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4bbc0ca240 0x7f4bbc156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 --2- 172.21.15.28:0/151151607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba80039b0 0x7f4bbc0cb830 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 >> 172.21.15.28:0/151151607 conn(0x7f4bbc09c4b0 msgr2=0x7f4bbc06c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 shutdown_connections 2023-12-16T21:57:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:48.964+0000 7f4bc3864640 1 -- 172.21.15.28:0/151151607 wait complete. 2023-12-16T21:57:48.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-16T21:57:48.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-16T21:57:48.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-16T21:57:48.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-16T21:57:48.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-16T21:57:48.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-16T21:57:48.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-16T21:57:48.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-16T21:57:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.148+0000 7fa6200d7640 1 Processor -- start 2023-12-16T21:57:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.148+0000 7fa6200d7640 1 -- start start 2023-12-16T21:57:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa6200d7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa6180ce6c0 0x7fa6180cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa6200d7640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181598e0 0x7fa618159d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa6200d7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61806e100 con 0x7fa6181500a0 2023-12-16T21:57:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa6200d7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61815a650 con 0x7fa6180ce6c0 2023-12-16T21:57:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa6200d7640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180cc740 con 0x7fa6181598e0 2023-12-16T21:57:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa61d672640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa6180ce6c0 0x7fa6180cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa61d672640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa6180ce6c0 0x7fa6180cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45916/0 (socket says 172.21.15.28:45916) 2023-12-16T21:57:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa61d672640 1 -- 172.21.15.28:0/97069047 learned_addr learned my addr 172.21.15.28:0/97069047 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa61e674640 1 --2- 172.21.15.28:0/97069047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181598e0 0x7fa618159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa61d672640 1 -- 172.21.15.28:0/97069047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181598e0 msgr2=0x7fa618159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa61d672640 1 --2- 172.21.15.28:0/97069047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181598e0 0x7fa618159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa61d672640 1 -- 172.21.15.28:0/97069047 >> v1:172.21.15.28:6789/0 conn(0x7fa6181500a0 legacy=0x7fa6180cdfb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:57:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.152+0000 7fa61d672640 1 -- 172.21.15.28:0/97069047 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180cc9e0 con 0x7fa6180ce6c0 2023-12-16T21:57:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7f8b57f6a640 1 Processor -- start 2023-12-16T21:57:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7fa61d672640 1 --2- 172.21.15.28:0/97069047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6180ce6c0 0x7fa6180cd7a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa614009870 tx=0x7fa61401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2685726cec218552 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7f8b57f6a640 1 -- start start 2023-12-16T21:57:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7fa61ce71640 1 -- 172.21.15.28:0/97069047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6140aa5f0 con 0x7fa6180ce6c0 2023-12-16T21:57:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7fa61ce71640 1 -- 172.21.15.28:0/97069047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6140aa750 con 0x7fa6180ce6c0 2023-12-16T21:57:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7fa61ce71640 1 -- 172.21.15.28:0/97069047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6140aaae0 con 0x7fa6180ce6c0 2023-12-16T21:57:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7fa6200d7640 1 -- 172.21.15.28:0/97069047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6180ce6c0 msgr2=0x7fa6180cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7fa6200d7640 1 --2- 172.21.15.28:0/97069047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6180ce6c0 0x7fa6180cd7a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa614009870 tx=0x7fa61401c4d0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7f8b57f6a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8b500cd7a0 0x7f8b50159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7f8b57f6a640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5006e100 con 0x7f8b500ce6c0 2023-12-16T21:57:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7f8b57f6a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b500cc740 con 0x7f8b501500a0 2023-12-16T21:57:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7f8b57f6a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5015a650 con 0x7f8b500cd7a0 2023-12-16T21:57:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7f8b55505640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b500ce6c0 0x7f8b501598e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51168/0 (socket says 172.21.15.28:51168) 2023-12-16T21:57:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.156+0000 7f8b55505640 1 -- 172.21.15.28:0/3404760644 learned_addr learned my addr 172.21.15.28:0/3404760644 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 -- 172.21.15.28:0/97069047 shutdown_connections 2023-12-16T21:57:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 --2- 172.21.15.28:0/97069047 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181598e0 0x7fa618159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 --2- 172.21.15.28:0/97069047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6180ce6c0 0x7fa6180cd7a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 -- 172.21.15.28:0/97069047 >> 172.21.15.28:0/97069047 conn(0x7fa6180aea50 msgr2=0x7fa6180722c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 -- 172.21.15.28:0/97069047 shutdown_connections 2023-12-16T21:57:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 -- 172.21.15.28:0/97069047 wait complete. 2023-12-16T21:57:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7f8b56507640 1 --2- 172.21.15.28:0/3404760644 >> v2:172.21.15.28:3300/0 conn(0x7f8b500cd7a0 0x7f8b50159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7f8b56507640 1 -- 172.21.15.28:0/3404760644 >> v1:172.21.15.122:6789/0 conn(0x7f8b501500a0 legacy=0x7f8b500cdfb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-16T21:57:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7f8b56507640 1 -- 172.21.15.28:0/3404760644 >> v1:172.21.15.28:6789/0 conn(0x7f8b500ce6c0 legacy=0x7f8b501598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7f8b56507640 1 -- 172.21.15.28:0/3404760644 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b500cc9e0 con 0x7f8b500cd7a0 2023-12-16T21:57:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 Processor -- start 2023-12-16T21:57:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 -- start start 2023-12-16T21:57:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6180ce6c0 0x7fa618155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181500a0 0x7fa6181583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6181598e0 0x7fa618147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618163060 con 0x7fa6181598e0 2023-12-16T21:57:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618156240 con 0x7fa6180ce6c0 2023-12-16T21:57:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa6200d7640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa618156510 con 0x7fa6181500a0 2023-12-16T21:57:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa61e674640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6181598e0 0x7fa618147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa61e674640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6181598e0 0x7fa618147850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45936/0 (socket says 172.21.15.28:45936) 2023-12-16T21:57:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa61e674640 1 -- 172.21.15.28:0/2051116001 learned_addr learned my addr 172.21.15.28:0/2051116001 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa61de73640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6180ce6c0 0x7fa618155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.160+0000 7fa61d672640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181500a0 0x7fa6181583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa61e674640 1 -- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181500a0 msgr2=0x7fa6181583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa61e674640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa6181500a0 0x7fa6181583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa61e674640 1 -- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6180ce6c0 msgr2=0x7fa618155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa61e674640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa6180ce6c0 0x7fa618155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa61e674640 1 -- 172.21.15.28:0/2051116001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6140094d0 con 0x7fa6181598e0 2023-12-16T21:57:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa61e674640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6181598e0 0x7fa618147850 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa61000efc0 tx=0x7fa61000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610009ba0 con 0x7fa6181598e0 2023-12-16T21:57:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61000f040 con 0x7fa6181598e0 2023-12-16T21:57:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6100146c0 con 0x7fa6181598e0 2023-12-16T21:57:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6181567f0 con 0x7fa6181598e0 2023-12-16T21:57:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa618156db0 con 0x7fa6181598e0 2023-12-16T21:57:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f1ce0ba6640 1 Processor -- start 2023-12-16T21:57:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f1ce0ba6640 1 -- start start 2023-12-16T21:57:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fae8247c640 1 Processor -- start 2023-12-16T21:57:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f8b56507640 1 --2- 172.21.15.28:0/3404760644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 0x7f8b50159ff0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b48009d90 tx=0x7f8b4801c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a901ba21bcc1acbb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f003a150640 1 Processor -- start 2023-12-16T21:57:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fae8247c640 1 -- start start 2023-12-16T21:57:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f003a150640 1 -- start start 2023-12-16T21:57:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fe2f7a8d640 1 Processor -- start 2023-12-16T21:57:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa618150480 con 0x7fa6181598e0 2023-12-16T21:57:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f1ce0ba6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1cdc1592f0 0x7f1cdc06c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f8b54d04640 1 -- 172.21.15.28:0/3404760644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b480a5840 con 0x7f8b500cd7a0 2023-12-16T21:57:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f8b54d04640 1 -- 172.21.15.28:0/3404760644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b480a59a0 con 0x7f8b500cd7a0 2023-12-16T21:57:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f8b54d04640 1 -- 172.21.15.28:0/3404760644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b480a5d30 con 0x7f8b500cd7a0 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fae8247c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c15a1b0 0x7fae7c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fae8247c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae7c0b2f40 0x7fae7c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fae8247c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7c06ee30 con 0x7fae7c06fc20 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fae8247c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7c0cdb60 con 0x7fae7c0b2f40 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fae8247c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c06f080 con 0x7fae7c15a1b0 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae80a19640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae7c0b2f40 0x7fae7c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f1cdb7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f1cdc158f10 0x7f1cdc0b2ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:58798/0 (socket says 172.21.15.28:58798) 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f1cdb7fe640 1 -- 172.21.15.28:0/648948180 learned_addr learned my addr 172.21.15.28:0/648948180 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7f1ce0ba6640 1 --2- 172.21.15.28:0/648948180 >> v2:172.21.15.28:3300/0 conn(0x7f1cdc0b3600 0x7f1cdc15d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f003a150640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f002c0368e0 0x7f002c04fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f003a150640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c052dd0 con 0x7f002c03a080 2023-12-16T21:57:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f003a150640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002c03ae70 con 0x7f002c0e3b70 2023-12-16T21:57:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f003a150640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c036cc0 con 0x7f002c0368e0 2023-12-16T21:57:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1cdaffd640 1 --2- 172.21.15.28:0/648948180 >> v2:172.21.15.122:3300/0 conn(0x7f1cdc1592f0 0x7f1cdc06c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae80a19640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae7c0b2f40 0x7fae7c06d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45938/0 (socket says 172.21.15.28:45938) 2023-12-16T21:57:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae80a19640 1 -- 172.21.15.28:0/1410361548 learned_addr learned my addr 172.21.15.28:0/1410361548 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae7bfff640 1 --2- 172.21.15.28:0/1410361548 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c15a1b0 0x7fae7c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1cdbfff640 1 --2- 172.21.15.28:0/648948180 >> v2:172.21.15.28:3300/0 conn(0x7f1cdc0b3600 0x7f1cdc15d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/648948180 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cdc0ccbb0 con 0x7f1cdc158f10 2023-12-16T21:57:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fe2f7a8d640 1 -- start start 2023-12-16T21:57:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f7a8d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe2e80e7490 0x7fe2e8054cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f7a8d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2e804fe80 0x7fe2e8052f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f7a8d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e8051a90 con 0x7fe2e8038180 2023-12-16T21:57:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f7a8d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e8053ff0 con 0x7fe2e804fe80 2023-12-16T21:57:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f7a8d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e804f980 con 0x7fe2e80e7490 2023-12-16T21:57:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f8b57f6a640 1 -- 172.21.15.28:0/3404760644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 msgr2=0x7f8b50159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f8b57f6a640 1 --2- 172.21.15.28:0/3404760644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 0x7f8b50159ff0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b48009d90 tx=0x7f8b4801c7a0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.173 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.164+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fa610014820 con 0x7fa6181598e0 2023-12-16T21:57:49.173 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa60e7fc640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa5fc066540 0x7fa5fc068a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.173 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa610084920 con 0x7fa6181598e0 2023-12-16T21:57:49.173 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa60e7fc640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fa5fc06e6e0 0x7fa5fc070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.173 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 --> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5fc0711b0 con 0x7fa5fc06e6e0 2023-12-16T21:57:49.173 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae80a19640 1 -- 172.21.15.28:0/1410361548 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c15a1b0 msgr2=0x7fae7c06d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae80a19640 1 --2- 172.21.15.28:0/1410361548 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c15a1b0 0x7fae7c06d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae80a19640 1 -- 172.21.15.28:0/1410361548 >> v1:172.21.15.28:6789/0 conn(0x7fae7c06fc20 legacy=0x7fae7c0714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae80a19640 1 -- 172.21.15.28:0/1410361548 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7c0cd7f0 con 0x7fae7c0b2f40 2023-12-16T21:57:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f0033fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f002c03a080 0x7f002c006200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51182/0 (socket says 172.21.15.28:51182) 2023-12-16T21:57:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f0033fff640 1 -- 172.21.15.28:0/2918067657 learned_addr learned my addr 172.21.15.28:0/2918067657 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f5829640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe2e80e7490 0x7fe2e8054cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f8b57f6a640 1 -- 172.21.15.28:0/3404760644 shutdown_connections 2023-12-16T21:57:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f8b57f6a640 1 --2- 172.21.15.28:0/3404760644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 0x7f8b50159ff0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f8b57f6a640 1 -- 172.21.15.28:0/3404760644 >> 172.21.15.28:0/3404760644 conn(0x7f8b500aea50 msgr2=0x7f8b500722c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/648948180 --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cdc0ce3e0 con 0x7f1cdc0b3600 2023-12-16T21:57:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/648948180 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cdc072160 con 0x7f1cdc1592f0 2023-12-16T21:57:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f8b57f6a640 1 -- 172.21.15.28:0/3404760644 shutdown_connections 2023-12-16T21:57:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f5829640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe2e80e7490 0x7fe2e8054cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:37694/0 (socket says 172.21.15.28:37694) 2023-12-16T21:57:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f5829640 1 -- 172.21.15.28:0/4251003620 learned_addr learned my addr 172.21.15.28:0/4251003620 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f5028640 1 --2- 172.21.15.28:0/4251003620 >> v2:172.21.15.28:3300/0 conn(0x7fe2e804fe80 0x7fe2e8052f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa610051d80 con 0x7fa6181598e0 2023-12-16T21:57:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa61d672640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fa5fc06e6e0 0x7fa5fc070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa61de73640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa5fc066540 0x7fa5fc068a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1cda7fc640 1 -- 172.21.15.28:0/648948180 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254013815 0 0) 0x7f1cdc0ccbb0 con 0x7f1cdc158f10 2023-12-16T21:57:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1cda7fc640 1 -- 172.21.15.28:0/648948180 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc4003530 con 0x7f1cdc158f10 2023-12-16T21:57:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f8b57f6a640 1 -- 172.21.15.28:0/3404760644 wait complete. 2023-12-16T21:57:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7249996640 1 Processor -- start 2023-12-16T21:57:49.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f5829640 1 -- 172.21.15.28:0/4251003620 >> v2:172.21.15.28:3300/0 conn(0x7fe2e804fe80 msgr2=0x7fe2e8052f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f5829640 1 --2- 172.21.15.28:0/4251003620 >> v2:172.21.15.28:3300/0 conn(0x7fe2e804fe80 0x7fe2e8052f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f5829640 1 -- 172.21.15.28:0/4251003620 >> v1:172.21.15.28:6789/0 conn(0x7fe2e8038180 legacy=0x7fe2e8053440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7249996640 1 -- start start 2023-12-16T21:57:49.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fe2f5829640 1 -- 172.21.15.28:0/4251003620 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e80485a0 con 0x7fe2e80e7490 2023-12-16T21:57:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1cdaffd640 1 -- 172.21.15.28:0/648948180 >> v2:172.21.15.28:3300/0 conn(0x7f1cdc0b3600 msgr2=0x7f1cdc15d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1cdaffd640 1 --2- 172.21.15.28:0/648948180 >> v2:172.21.15.28:3300/0 conn(0x7f1cdc0b3600 0x7f1cdc15d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1cdaffd640 1 -- 172.21.15.28:0/648948180 >> v1:172.21.15.122:6789/0 conn(0x7f1cdc158f10 legacy=0x7f1cdc0b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f1cdaffd640 1 -- 172.21.15.28:0/648948180 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc072400 con 0x7f1cdc1592f0 2023-12-16T21:57:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa61de73640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa5fc066540 0x7fa5fc068a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa608007fd0 tx=0x7fa608009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fa61d672640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fa5fc06e6e0 0x7fa5fc070b00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7249996640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72440ad680 0x7f724415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7249996640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f724406e2d0 0x7f72440b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7243fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f724406e2d0 0x7f72440b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7249996640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72440ada60 con 0x7f7244071ae0 2023-12-16T21:57:49.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f8b57f6a640 1 Processor -- start 2023-12-16T21:57:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7249996640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72440b2080 con 0x7f724406e2d0 2023-12-16T21:57:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7249996640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72440b2360 con 0x7f72440ad680 2023-12-16T21:57:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7243fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f724406e2d0 0x7f72440b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45972/0 (socket says 172.21.15.28:45972) 2023-12-16T21:57:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7243fff640 1 -- 172.21.15.28:0/1688921367 learned_addr learned my addr 172.21.15.28:0/1688921367 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7243fff640 1 -- 172.21.15.28:0/1688921367 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72440ad680 msgr2=0x7f724415dfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:57:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7243fff640 1 --2- 172.21.15.28:0/1688921367 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72440ad680 0x7f724415dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7243fff640 1 -- 172.21.15.28:0/1688921367 >> v1:172.21.15.28:6789/0 conn(0x7f7244071ae0 legacy=0x7f724406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7fae80a19640 1 --2- 172.21.15.28:0/1410361548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c0b2f40 0x7fae7c06d840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fae70009870 tx=0x7fae7001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c17c071668dbcc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f7243fff640 1 -- 172.21.15.28:0/1688921367 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72440b2640 con 0x7f724406e2d0 2023-12-16T21:57:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f00337fe640 1 --2- 172.21.15.28:0/2918067657 >> v2:172.21.15.122:3300/0 conn(0x7f002c0368e0 0x7f002c04fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f00337fe640 1 -- 172.21.15.28:0/2918067657 >> v1:172.21.15.122:6789/0 conn(0x7f002c0e3b70 legacy=0x7f002c0512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f00337fe640 1 -- 172.21.15.28:0/2918067657 >> v1:172.21.15.28:6789/0 conn(0x7f002c03a080 legacy=0x7f002c006200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.168+0000 7f00337fe640 1 -- 172.21.15.28:0/2918067657 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c04ff60 con 0x7f002c0368e0 2023-12-16T21:57:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fae7affd640 1 -- 172.21.15.28:0/1410361548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae700aa640 con 0x7fae7c0b2f40 2023-12-16T21:57:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fae7affd640 1 -- 172.21.15.28:0/1410361548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae700aa7a0 con 0x7fae7c0b2f40 2023-12-16T21:57:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 -- start start 2023-12-16T21:57:49.184 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 <== osd.5 v2:172.21.15.122:6816/1068560627 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa5fc0711b0 con 0x7fa5fc06e6e0 2023-12-16T21:57:49.184 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1cdaffd640 1 --2- 172.21.15.28:0/648948180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cdc1592f0 0x7f1cdc06c520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc15cbc0 tx=0x7f1cd001c700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=953519b2cd826b56 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.184 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fae8247c640 1 -- 172.21.15.28:0/1410361548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c0b2f40 msgr2=0x7fae7c06d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.184 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fae8247c640 1 --2- 172.21.15.28:0/1410361548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c0b2f40 0x7fae7c06d840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fae70009870 tx=0x7fae7001c500 comp rx=0 tx=0).stop 2023-12-16T21:57:49.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1cda7fc640 1 -- 172.21.15.28:0/648948180 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00a57e0 con 0x7f1cdc1592f0 2023-12-16T21:57:49.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 0x7f8b50153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8b500ce6c0 0x7f8b501579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b501500a0 0x7f8b5014aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500cdc10 con 0x7f8b500cd7a0 2023-12-16T21:57:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b50148fb0 con 0x7f8b501500a0 2023-12-16T21:57:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b50157ef0 con 0x7f8b500ce6c0 2023-12-16T21:57:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1cda7fc640 1 -- 172.21.15.28:0/648948180 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd00a5940 con 0x7f1cdc1592f0 2023-12-16T21:57:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1cda7fc640 1 -- 172.21.15.28:0/648948180 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00a5cd0 con 0x7f1cdc1592f0 2023-12-16T21:57:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 0x7f8b50153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f00337fe640 1 --2- 172.21.15.28:0/2918067657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f002c0368e0 0x7f002c04fa20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f00200097c0 tx=0x7f002001c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=723c3625551e94c3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 0x7f8b50153d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45976/0 (socket says 172.21.15.28:45976) 2023-12-16T21:57:49.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 -- 172.21.15.28:0/2348267755 learned_addr learned my addr 172.21.15.28:0/2348267755 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55505640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8b500ce6c0 0x7f8b501579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/648948180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cdc1592f0 msgr2=0x7f1cdc06c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 --2- 172.21.15.28:0/648948180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cdc1592f0 0x7f1cdc06c520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc15cbc0 tx=0x7f1cd001c700 comp rx=0 tx=0).stop 2023-12-16T21:57:49.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b56507640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b501500a0 0x7f8b5014aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/648948180 shutdown_connections 2023-12-16T21:57:49.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 --2- 172.21.15.28:0/648948180 >> v2:172.21.15.28:3300/0 conn(0x7f1cdc0b3600 0x7f1cdc15d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 --2- 172.21.15.28:0/648948180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cdc1592f0 0x7f1cdc06c520 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/648948180 >> 172.21.15.28:0/648948180 conn(0x7f1cdc0b0ae0 msgr2=0x7f1cdc0b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 -- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8b500ce6c0 msgr2=0x7f8b501579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8b500ce6c0 0x7f8b501579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 -- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b501500a0 msgr2=0x7f8b5014aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b501500a0 0x7f8b5014aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f7243fff640 1 --2- 172.21.15.28:0/1688921367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 0x7f72440b3bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7238009870 tx=0x7f723801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bef9dd751099729 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f00327fc640 1 -- 172.21.15.28:0/2918067657 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200ab600 con 0x7f002c0368e0 2023-12-16T21:57:49.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f00327fc640 1 -- 172.21.15.28:0/2918067657 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00200a7050 con 0x7f002c0368e0 2023-12-16T21:57:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 -- 172.21.15.28:0/2348267755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b48009a70 con 0x7f8b500cd7a0 2023-12-16T21:57:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/648948180 shutdown_connections 2023-12-16T21:57:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b55d06640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 0x7f8b50153d00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000b410 tx=0x7f8b4000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f5829640 1 --2- 172.21.15.28:0/4251003620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e80e7490 0x7fe2e8054cd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009870 tx=0x7fe2ec01ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6c58a080abf5ab1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f72437fe640 1 -- 172.21.15.28:0/1688921367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72380aa430 con 0x7f724406e2d0 2023-12-16T21:57:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f72437fe640 1 -- 172.21.15.28:0/1688921367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72380aa590 con 0x7f724406e2d0 2023-12-16T21:57:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40011830 con 0x7f8b500cd7a0 2023-12-16T21:57:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40011990 con 0x7f8b500cd7a0 2023-12-16T21:57:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 -- 172.21.15.28:0/2348267755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b501542e0 con 0x7f8b500cd7a0 2023-12-16T21:57:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f00327fc640 1 -- 172.21.15.28:0/2918067657 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00200ab980 con 0x7f002c0368e0 2023-12-16T21:57:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f003a150640 1 -- 172.21.15.28:0/2918067657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f002c0368e0 msgr2=0x7f002c04fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f003a150640 1 --2- 172.21.15.28:0/2918067657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f002c0368e0 0x7f002c04fa20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f00200097c0 tx=0x7f002001c430 comp rx=0 tx=0).stop 2023-12-16T21:57:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f003a150640 1 -- 172.21.15.28:0/2918067657 shutdown_connections 2023-12-16T21:57:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f003a150640 1 --2- 172.21.15.28:0/2918067657 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f002c0368e0 0x7f002c04fa20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f003a150640 1 -- 172.21.15.28:0/2918067657 >> 172.21.15.28:0/2918067657 conn(0x7f002c01e6a0 msgr2=0x7f002c0553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f4827640 1 -- 172.21.15.28:0/4251003620 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec0a89f0 con 0x7fe2e80e7490 2023-12-16T21:57:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f4827640 1 -- 172.21.15.28:0/4251003620 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec0a8b50 con 0x7fe2e80e7490 2023-12-16T21:57:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f4827640 1 -- 172.21.15.28:0/4251003620 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec0b0440 con 0x7fe2e80e7490 2023-12-16T21:57:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4251003620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e80e7490 msgr2=0x7fe2e8054cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f7a8d640 1 --2- 172.21.15.28:0/4251003620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e80e7490 0x7fe2e8054cd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009870 tx=0x7fe2ec01ecf0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f7249996640 1 -- 172.21.15.28:0/1688921367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 msgr2=0x7f72440b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f7249996640 1 --2- 172.21.15.28:0/1688921367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 0x7f72440b3bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7238009870 tx=0x7f723801c4f0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/648948180 wait complete. 2023-12-16T21:57:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 Processor -- start 2023-12-16T21:57:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b57f6a640 1 -- 172.21.15.28:0/2348267755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b501548a0 con 0x7f8b500cd7a0 2023-12-16T21:57:49.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4251003620 shutdown_connections 2023-12-16T21:57:49.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f7a8d640 1 --2- 172.21.15.28:0/4251003620 >> v2:172.21.15.28:3300/0 conn(0x7fe2e804fe80 0x7fe2e8052f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f7a8d640 1 --2- 172.21.15.28:0/4251003620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e80e7490 0x7fe2e8054cd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4251003620 >> 172.21.15.28:0/4251003620 conn(0x7fe2e8055330 msgr2=0x7fe2e8055740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40011d40 con 0x7f8b500cd7a0 2023-12-16T21:57:49.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b57f6a640 1 -- 172.21.15.28:0/2348267755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b500cd280 con 0x7f8b500cd7a0 2023-12-16T21:57:49.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f7249996640 1 -- 172.21.15.28:0/1688921367 shutdown_connections 2023-12-16T21:57:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f7249996640 1 --2- 172.21.15.28:0/1688921367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 0x7f72440b3bd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f7249996640 1 --2- 172.21.15.28:0/1688921367 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72440ad680 0x7f724415dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f7249996640 1 -- 172.21.15.28:0/1688921367 >> 172.21.15.28:0/1688921367 conn(0x7f72440b4500 msgr2=0x7f72440b4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f7249996640 1 -- 172.21.15.28:0/1688921367 shutdown_connections 2023-12-16T21:57:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f7249996640 1 -- 172.21.15.28:0/1688921367 wait complete. 2023-12-16T21:57:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.172+0000 7f1ce0ba6640 1 -- start start 2023-12-16T21:57:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1ce0ba6640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1cdc0b3600 0x7f1cdc1426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1ce0ba6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc158f10 0x7f1cdc14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1ce0ba6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cdc14d4e0 0x7f1cdc145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1ce0ba6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc15a690 con 0x7f1cdc158f10 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1ce0ba6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc142be0 con 0x7f1cdc14d4e0 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1ce0ba6640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc159a80 con 0x7f1cdc0b3600 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc158f10 0x7f1cdc14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc158f10 0x7f1cdc14cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45990/0 (socket says 172.21.15.28:45990) 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 -- 172.21.15.28:0/1844159768 learned_addr learned my addr 172.21.15.28:0/1844159768 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdbfff640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cdc14d4e0 0x7f1cdc145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4251003620 shutdown_connections 2023-12-16T21:57:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 -- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1cdc0b3600 msgr2=0x7f1cdc1426a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-16T21:57:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1cdc0b3600 0x7f1cdc1426a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 -- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cdc14d4e0 msgr2=0x7f1cdc145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cdc14d4e0 0x7f1cdc145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 -- 172.21.15.28:0/1844159768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd000a4d0 con 0x7f1cdc158f10 2023-12-16T21:57:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cdaffd640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc158f10 0x7f1cdc14cfa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1cd001a330 tx=0x7f1cd00194a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f8b4001a400 con 0x7f8b500cd7a0 2023-12-16T21:57:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f7249996640 1 Processor -- start 2023-12-16T21:57:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fae8247c640 1 -- 172.21.15.28:0/1410361548 shutdown_connections 2023-12-16T21:57:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fae8247c640 1 --2- 172.21.15.28:0/1410361548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c0b2f40 0x7fae7c06d840 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fae8247c640 1 --2- 172.21.15.28:0/1410361548 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c15a1b0 0x7fae7c06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fae8247c640 1 -- 172.21.15.28:0/1410361548 >> 172.21.15.28:0/1410361548 conn(0x7fae7c0cfe90 msgr2=0x7fae7c0d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f7249996640 1 -- start start 2023-12-16T21:57:49.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b467fc640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8b340664c0 0x7f8b34068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b400844e0 con 0x7f8b500cd7a0 2023-12-16T21:57:49.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b467fc640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f8b3406e570 0x7f8b34070990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 --> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b34071040 con 0x7f8b3406e570 2023-12-16T21:57:49.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b40084980 con 0x7f8b500cd7a0 2023-12-16T21:57:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fae8247c640 1 -- 172.21.15.28:0/1410361548 shutdown_connections 2023-12-16T21:57:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fae8247c640 1 -- 172.21.15.28:0/1410361548 wait complete. 2023-12-16T21:57:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b55505640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8b340664c0 0x7f8b34068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f8b55505640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8b340664c0 0x7f8b34068980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009fd0 tx=0x7f8b4c00b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4251003620 wait complete. 2023-12-16T21:57:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fe2f7a8d640 1 Processor -- start 2023-12-16T21:57:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00ad440 con 0x7f1cdc158f10 2023-12-16T21:57:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc159d60 con 0x7f1cdc158f10 2023-12-16T21:57:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc15a320 con 0x7f1cdc158f10 2023-12-16T21:57:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd00ad5a0 con 0x7f1cdc158f10 2023-12-16T21:57:49.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00ad910 con 0x7f1cdc158f10 2023-12-16T21:57:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f8b56507640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f8b3406e570 0x7f8b34070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae8247c640 1 Processor -- start 2023-12-16T21:57:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 -- 172.21.15.28:0/2918067657 shutdown_connections 2023-12-16T21:57:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 -- 172.21.15.28:0/2918067657 wait complete. 2023-12-16T21:57:49.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f8b56507640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f8b3406e570 0x7f8b34070990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.176+0000 7fe2f7a8d640 1 -- start start 2023-12-16T21:57:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f7a8d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e8038180 0x7fe2e80e6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f7a8d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e804fe80 0x7fe2e80d5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f7a8d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe2e80e7490 0x7fe2e80d55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f7a8d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80caa80 con 0x7fe2e8038180 2023-12-16T21:57:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f7a8d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80c9bc0 con 0x7fe2e804fe80 2023-12-16T21:57:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f7a8d640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80c9e00 con 0x7fe2e80e7490 2023-12-16T21:57:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1cbc000fd0 con 0x7f1cdc158f10 2023-12-16T21:57:49.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae8247c640 1 -- start start 2023-12-16T21:57:49.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae8247c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c0b2f40 0x7fae7c148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae8247c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae7c15a1b0 0x7fae7c144a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae8247c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c14e2d0 0x7fae7c149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae8247c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c14f6c0 con 0x7fae7c14e2d0 2023-12-16T21:57:49.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae8247c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c14eab0 con 0x7fae7c15a1b0 2023-12-16T21:57:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae8247c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c14ed90 con 0x7fae7c0b2f40 2023-12-16T21:57:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c14e2d0 0x7fae7c149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c14e2d0 0x7fae7c149260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45994/0 (socket says 172.21.15.28:45994) 2023-12-16T21:57:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 -- 172.21.15.28:0/761055090 learned_addr learned my addr 172.21.15.28:0/761055090 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 <== osd.3 v2:172.21.15.28:6800/4021281394 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8b34071040 con 0x7f8b3406e570 2023-12-16T21:57:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae7b7fe640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae7c15a1b0 0x7fae7c144a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 Processor -- start 2023-12-16T21:57:49.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 -- start start 2023-12-16T21:57:49.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f602a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe2e80e7490 0x7fe2e80d55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f602a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe2e80e7490 0x7fe2e80d55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:59954/0 (socket says 172.21.15.28:59954) 2023-12-16T21:57:49.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f602a640 1 -- 172.21.15.28:0/4026722393 learned_addr learned my addr 172.21.15.28:0/4026722393 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f5028640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e804fe80 0x7fe2e80d5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae7bfff640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c0b2f40 0x7fae7c148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 -- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c0b2f40 msgr2=0x7fae7c148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae7c0b2f40 0x7fae7c148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 -- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae7c15a1b0 msgr2=0x7fae7c144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae7c15a1b0 0x7fae7c144a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 -- 172.21.15.28:0/761055090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae700094d0 con 0x7fae7c14e2d0 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f002c0368e0 0x7f002c0e2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f002c03a080 0x7f002c188550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f002c0e3b70 0x7f002c0e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0ce930 con 0x7f002c03a080 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0cec10 con 0x7f002c0368e0 2023-12-16T21:57:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f003a150640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0ceef0 con 0x7f002c0e3b70 2023-12-16T21:57:49.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0033fff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f002c0e3b70 0x7f002c0e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7249996640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 0x7f72441453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fae80a19640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c14e2d0 0x7fae7c149260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fae700ac040 tx=0x7fae7001ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7249996640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7244071ae0 0x7f7244155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0033fff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f002c0e3b70 0x7f002c0e0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:59974/0 (socket says 172.21.15.28:59974) 2023-12-16T21:57:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0033fff640 1 -- 172.21.15.28:0/10926317 learned_addr learned my addr 172.21.15.28:0/10926317 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0032ffd640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f002c03a080 0x7f002c188550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f5829640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e8038180 0x7fe2e80e6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7249996640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72440ad680 0x7f7244155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7249996640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72441530e0 con 0x7f724406e2d0 2023-12-16T21:57:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f00337fe640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f002c0368e0 0x7f002c0e2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7249996640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72441533c0 con 0x7f7244071ae0 2023-12-16T21:57:49.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7249996640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72441536a0 con 0x7f72440ad680 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f5028640 1 -- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe2e80e7490 msgr2=0x7fe2e80d55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f5028640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe2e80e7490 0x7fe2e80d55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f5028640 1 -- 172.21.15.28:0/4026722393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e8038180 msgr2=0x7fe2e80e6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f5028640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e8038180 0x7fe2e80e6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7fe2f5028640 1 -- 172.21.15.28:0/4026722393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec0094d0 con 0x7fe2e804fe80 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0032ffd640 1 -- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f002c0e3b70 msgr2=0x7f002c0e0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0032ffd640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f002c0e3b70 0x7f002c0e0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0032ffd640 1 -- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f002c0368e0 msgr2=0x7f002c0e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0032ffd640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f002c0368e0 0x7f002c0e2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f0032ffd640 1 -- 172.21.15.28:0/10926317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020009420 con 0x7f002c03a080 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fe2f5028640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e804fe80 0x7fe2e80d5070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000e870 tx=0x7fe2e000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00095d0 con 0x7fe2e804fe80 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e000c940 con 0x7fe2e804fe80 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000ccb0 con 0x7fe2e804fe80 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae700aa640 con 0x7fae7c14e2d0 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae700ba070 con 0x7fae7c14e2d0 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae700a9c40 con 0x7fae7c14e2d0 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c14f070 con 0x7fae7c14e2d0 2023-12-16T21:57:49.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c14e780 con 0x7fae7c14e2d0 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7248994640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 0x7f72441453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7248994640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 0x7f72441453e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46022/0 (socket says 172.21.15.28:46022) 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7248994640 1 -- 172.21.15.28:0/301676411 learned_addr learned my addr 172.21.15.28:0/301676411 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7243fff640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7244071ae0 0x7f7244155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.180+0000 7f7249195640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72440ad680 0x7f7244155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f7248994640 1 -- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72440ad680 msgr2=0x7f7244155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f7248994640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72440ad680 0x7f7244155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f7248994640 1 -- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7244071ae0 msgr2=0x7f7244155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f7248994640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7244071ae0 0x7f7244155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f7248994640 1 -- 172.21.15.28:0/301676411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72380094d0 con 0x7f724406e2d0 2023-12-16T21:57:49.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1cd00b4070 con 0x7f1cdc158f10 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cc3fff640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1cc8066470 0x7f1cc8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cd011ebb0 con 0x7f1cdc158f10 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cc3fff640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f1cc806e610 0x7f1cc8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 --> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cc80710e0 con 0x7f1cc806e610 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cd011ef20 con 0x7f1cdc158f10 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cdbfff640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f1cc806e610 0x7f1cc8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cdb7fe640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1cc8066470 0x7f1cc8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cdbfff640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f1cc806e610 0x7f1cc8070a30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f0032ffd640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f002c03a080 0x7f002c188550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0028000f70 tx=0x7f002800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00280052d0 con 0x7f002c03a080 2023-12-16T21:57:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f7248994640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 0x7f72441453e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f723400b4a0 tx=0x7f723400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f1cdb7fe640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1cc8066470 0x7f1cc8068930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc006120 tx=0x7f1ccc005b80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002c0cf1d0 con 0x7f002c03a080 2023-12-16T21:57:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c04bee0 con 0x7f002c03a080 2023-12-16T21:57:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0028005430 con 0x7f002c03a080 2023-12-16T21:57:49.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0028003840 con 0x7f002c03a080 2023-12-16T21:57:49.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f002c050da0 con 0x7f002c03a080 2023-12-16T21:57:49.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e80ca0e0 con 0x7fe2e804fe80 2023-12-16T21:57:49.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e80ca6a0 con 0x7fe2e804fe80 2023-12-16T21:57:49.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae7c1458d0 con 0x7fae7c14e2d0 2023-12-16T21:57:49.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234010610 con 0x7f724406e2d0 2023-12-16T21:57:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7234010770 con 0x7f724406e2d0 2023-12-16T21:57:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.184+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7234010b60 con 0x7f724406e2d0 2023-12-16T21:57:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 <== osd.6 v2:172.21.15.122:6824/1879423542 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1cc80710e0 con 0x7f1cc806e610 2023-12-16T21:57:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f00280039a0 con 0x7f002c03a080 2023-12-16T21:57:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f001ffff640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00180664f0 0x7f00180689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0028083600 con 0x7f002c03a080 2023-12-16T21:57:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f001ffff640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7f001806e710 0x7f0018070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 --> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00180711e0 con 0x7f001806e710 2023-12-16T21:57:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0028083aa0 con 0x7f002c03a080 2023-12-16T21:57:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f00337fe640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00180664f0 0x7f00180689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fae700a8050 con 0x7fae7c14e2d0 2023-12-16T21:57:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae78ff9640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae600664f0 0x7fae600689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.219 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f0033fff640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7f001806e710 0x7f0018070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae70121c50 con 0x7fae7c14e2d0 2023-12-16T21:57:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae78ff9640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7fae6006e690 0x7fae60070ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 --> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae60071160 con 0x7fae6006e690 2023-12-16T21:57:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae7b7fe640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7fae6006e690 0x7fae60070ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae7bfff640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae600664f0 0x7fae600689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae7b7fe640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7fae6006e690 0x7fae60070ab0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f0033fff640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7f001806e710 0x7f0018070b30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7f00337fe640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00180664f0 0x7f00180689b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00200ad000 tx=0x7f0020008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 <== osd.2 v2:172.21.15.28:6807/451272560 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fae60071160 con 0x7fae6006e690 2023-12-16T21:57:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.188+0000 7fae7bfff640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae600664f0 0x7fae600689b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae7400b240 tx=0x7fae7400f040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9f95e640 1 Processor -- start 2023-12-16T21:57:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 <== osd.7 v2:172.21.15.122:6808/4051779013 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f00180711e0 con 0x7f001806e710 2023-12-16T21:57:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9f95e640 1 -- start start 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2e8048b60 con 0x7fe2e804fe80 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe2e00059a0 con 0x7fe2e804fe80 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2e67fc640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe2d40664c0 0x7fe2d4068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2e00846c0 con 0x7fe2e804fe80 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2e67fc640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7fe2d406e6e0 0x7fe2d4070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 --> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2d40711b0 con 0x7fe2d406e6e0 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2e0084a30 con 0x7fe2e804fe80 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2f5829640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe2d40664c0 0x7fe2d4068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2f602a640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7fe2d406e6e0 0x7fe2d4070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae700aab30 con 0x7fae7c14e2d0 2023-12-16T21:57:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9f95e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8d9815b230 0x7f8d980cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2f602a640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7fe2d406e6e0 0x7fe2d4070b00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2f5829640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe2d40664c0 0x7fe2d4068980 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec0b2000 tx=0x7fe2ec009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9f95e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d980b0d80 0x7f8d9815ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9f95e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980cc320 con 0x7f8d980cc7b0 2023-12-16T21:57:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9f95e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d980b11d0 con 0x7f8d980b0d80 2023-12-16T21:57:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9f95e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d980cfbf0 con 0x7f8d9815b230 2023-12-16T21:57:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9d6fa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8d9815b230 0x7f8d980cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9d6fa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8d9815b230 0x7f8d980cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:37764/0 (socket says 172.21.15.28:37764) 2023-12-16T21:57:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9d6fa640 1 -- 172.21.15.28:0/1720112349 learned_addr learned my addr 172.21.15.28:0/1720112349 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9d6fa640 1 -- 172.21.15.28:0/1720112349 >> v2:172.21.15.28:3300/0 conn(0x7f8d980b0d80 msgr2=0x7f8d9815ec00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:57:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9d6fa640 1 --2- 172.21.15.28:0/1720112349 >> v2:172.21.15.28:3300/0 conn(0x7f8d980b0d80 0x7f8d9815ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9d6fa640 1 -- 172.21.15.28:0/1720112349 >> v1:172.21.15.122:6789/0 conn(0x7f8d980cc7b0 legacy=0x7f8d980b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7f8d9d6fa640 1 -- 172.21.15.28:0/1720112349 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980cbcd0 con 0x7f8d9815b230 2023-12-16T21:57:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.192+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 <== osd.1 v2:172.21.15.28:6824/3615286709 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe2d40711b0 con 0x7fe2d406e6e0 2023-12-16T21:57:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9d6fa640 1 --2- 172.21.15.28:0/1720112349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d9815b230 0x7f8d980cf510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009870 tx=0x7f8d9401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6994a67b9668926 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d8ffff640 1 -- 172.21.15.28:0/1720112349 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d940ab600 con 0x7f8d9815b230 2023-12-16T21:57:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d8ffff640 1 -- 172.21.15.28:0/1720112349 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d940a7050 con 0x7f8d9815b230 2023-12-16T21:57:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d8ffff640 1 -- 172.21.15.28:0/1720112349 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d940ab980 con 0x7f8d9815b230 2023-12-16T21:57:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 --> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6180693d0 con 0x7fa5fc06e6e0 2023-12-16T21:57:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9f95e640 1 -- 172.21.15.28:0/1720112349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d9815b230 msgr2=0x7f8d980cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9f95e640 1 --2- 172.21.15.28:0/1720112349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d9815b230 0x7f8d980cf510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009870 tx=0x7f8d9401c450 comp rx=0 tx=0).stop 2023-12-16T21:57:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9f95e640 1 -- 172.21.15.28:0/1720112349 shutdown_connections 2023-12-16T21:57:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9f95e640 1 --2- 172.21.15.28:0/1720112349 >> v2:172.21.15.28:3300/0 conn(0x7f8d980b0d80 0x7f8d9815ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9f95e640 1 --2- 172.21.15.28:0/1720112349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d9815b230 0x7f8d980cf510 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9f95e640 1 -- 172.21.15.28:0/1720112349 >> 172.21.15.28:0/1720112349 conn(0x7f8d980d0f30 msgr2=0x7f8d980d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7fa60e7fc640 1 -- 172.21.15.28:0/2051116001 <== osd.5 v2:172.21.15.122:6816/1068560627 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa6180693d0 con 0x7fa5fc06e6e0 2023-12-16T21:57:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7244153980 con 0x7f724406e2d0 2023-12-16T21:57:49.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9f95e640 1 -- 172.21.15.28:0/1720112349 shutdown_connections 2023-12-16T21:57:49.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f8d9f95e640 1 -- 172.21.15.28:0/1720112349 wait complete. 2023-12-16T21:57:49.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fa5fc06e6e0 msgr2=0x7fa5fc070b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7fa6200d7640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fa5fc06e6e0 0x7fa5fc070b00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7244145920 con 0x7f724406e2d0 2023-12-16T21:57:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa5fc066540 msgr2=0x7fa5fc068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7fa6200d7640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa5fc066540 0x7fa5fc068a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa608007fd0 tx=0x7fa608009040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6181598e0 msgr2=0x7fa618147850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.196+0000 7fa6200d7640 1 --2- 172.21.15.28:0/2051116001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6181598e0 0x7fa618147850 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa61000efc0 tx=0x7fa61000c490 comp rx=0 tx=0).stop 2023-12-16T21:57:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 Processor -- start 2023-12-16T21:57:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 -- start start 2023-12-16T21:57:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7244153530 con 0x7f724406e2d0 2023-12-16T21:57:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7234010cc0 con 0x7f724406e2d0 2023-12-16T21:57:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7fa61e674640 1 -- 172.21.15.28:0/2051116001 reap_dead start 2023-12-16T21:57:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 shutdown_connections 2023-12-16T21:57:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 >> 172.21.15.28:0/2051116001 conn(0x7fa6180aea50 msgr2=0x7fa618148940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 shutdown_connections 2023-12-16T21:57:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7fa6200d7640 1 -- 172.21.15.28:0/2051116001 wait complete. 2023-12-16T21:57:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b0d80 0x7f8d980c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d980cc7b0 0x7f8d980b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d9815b230 0x7f8d980b5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d980b4dd0 con 0x7f8d980b0d80 2023-12-16T21:57:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d980c78f0 con 0x7f8d9815b230 2023-12-16T21:57:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b0d80 0x7f8d980c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b0d80 0x7f8d980c73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46038/0 (socket says 172.21.15.28:46038) 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 -- 172.21.15.28:0/2457083161 learned_addr learned my addr 172.21.15.28:0/2457083161 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9815e2e0 con 0x7f8d980cc7b0 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9defb640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d9815b230 0x7f8d980b5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9cef9640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d980cc7b0 0x7f8d980b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 -- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d980cc7b0 msgr2=0x7f8d980b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d980cc7b0 0x7f8d980b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 -- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d9815b230 msgr2=0x7f8d980b5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d9815b230 0x7f8d980b5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 -- 172.21.15.28:0/2457083161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d940094d0 con 0x7f8d980b0d80 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9cef9640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d980cc7b0 0x7f8d980b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9d6fa640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b0d80 0x7f8d980c73b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d9401a080 tx=0x7f8d9401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d940b5aa0 con 0x7f8d980b0d80 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f72417fa640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f72300664f0 0x7f72300689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7234083af0 con 0x7f724406e2d0 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f72417fa640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f723006e710 0x7f7230070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 --> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72300711e0 con 0x7f723006e710 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72340882a0 con 0x7f724406e2d0 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f7243fff640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f72300664f0 0x7f72300689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f7249195640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f723006e710 0x7f7230070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f7243fff640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f72300664f0 0x7f72300689b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f72380ae000 tx=0x7f72380b5040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f7249195640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f723006e710 0x7f7230070b30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d940b9070 con 0x7f8d980b0d80 2023-12-16T21:57:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d940b5e60 con 0x7f8d980b0d80 2023-12-16T21:57:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9815e5c0 con 0x7f8d980b0d80 2023-12-16T21:57:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.200+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d980b52c0 con 0x7f8d980b0d80 2023-12-16T21:57:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8d980674d0 con 0x7f8d980b0d80 2023-12-16T21:57:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f8d940c8450 con 0x7f8d980b0d80 2023-12-16T21:57:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 <== osd.4 v2:172.21.15.122:6800/1281108383 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f72300711e0 con 0x7f723006e710 2023-12-16T21:57:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d8dffb640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d7c066470 0x7f8d7c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d94121f80 con 0x7f8d980b0d80 2023-12-16T21:57:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d8dffb640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7f8d7c06e610 0x7f8d7c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 --> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d7c0710e0 con 0x7f8d7c06e610 2023-12-16T21:57:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d940ef480 con 0x7f8d980b0d80 2023-12-16T21:57:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d9cef9640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d7c066470 0x7f8d7c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d9defb640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7f8d7c06e610 0x7f8d7c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d9defb640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7f8d7c06e610 0x7f8d7c070a30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.204+0000 7f8d9cef9640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d7c066470 0x7f8d7c068930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8d9806e980 tx=0x7f8d88009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.208+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 <== osd.0 v2:172.21.15.28:6810/177892190 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8d7c0710e0 con 0x7f8d7c06e610 2023-12-16T21:57:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 --> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2e8048fd0 con 0x7fe2d406e6e0 2023-12-16T21:57:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2e67fc640 1 -- 172.21.15.28:0/4026722393 <== osd.1 v2:172.21.15.28:6824/3615286709 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe2f004f120 con 0x7fe2d406e6e0 2023-12-16T21:57:49.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7fe2d406e6e0 msgr2=0x7fe2d4070b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7fe2d406e6e0 0x7fe2d4070b00 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe2d40664c0 msgr2=0x7fe2d4068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe2d40664c0 0x7fe2d4068980 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec0b2000 tx=0x7fe2ec009040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e804fe80 msgr2=0x7fe2e80d5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 --2- 172.21.15.28:0/4026722393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe2e804fe80 0x7fe2e80d5070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000e870 tx=0x7fe2e000ed30 comp rx=0 tx=0).stop 2023-12-16T21:57:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f602a640 1 -- 172.21.15.28:0/4026722393 reap_dead start 2023-12-16T21:57:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 shutdown_connections 2023-12-16T21:57:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 >> 172.21.15.28:0/4026722393 conn(0x7fe2e8055330 msgr2=0x7fe2e8006330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 shutdown_connections 2023-12-16T21:57:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7fe2f7a8d640 1 -- 172.21.15.28:0/4026722393 wait complete. 2023-12-16T21:57:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.216+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 --> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f002c050ac0 con 0x7f001806e710 2023-12-16T21:57:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f001ffff640 1 -- 172.21.15.28:0/10926317 <== osd.7 v2:172.21.15.122:6808/4051779013 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f003404f180 con 0x7f001806e710 2023-12-16T21:57:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7f001806e710 msgr2=0x7f0018070b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7f001806e710 0x7f0018070b30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00180664f0 msgr2=0x7f00180689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00180664f0 0x7f00180689b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f00200ad000 tx=0x7f0020008040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f002c03a080 msgr2=0x7f002c188550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 --2- 172.21.15.28:0/10926317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f002c03a080 0x7f002c188550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0028000f70 tx=0x7f002800c890 comp rx=0 tx=0).stop 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f0033fff640 1 -- 172.21.15.28:0/10926317 reap_dead start 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 shutdown_connections 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 >> 172.21.15.28:0/10926317 conn(0x7f002c01e6a0 msgr2=0x7f002c007b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 shutdown_connections 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f003a150640 1 -- 172.21.15.28:0/10926317 wait complete. 2023-12-16T21:57:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 --> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1cbc002cf0 con 0x7f1cc806e610 2023-12-16T21:57:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.220+0000 7f1cc3fff640 1 -- 172.21.15.28:0/1844159768 <== osd.6 v2:172.21.15.122:6824/1879423542 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1cbc002cf0 con 0x7f1cc806e610 2023-12-16T21:57:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f1cc806e610 msgr2=0x7f1cc8070a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f1cc806e610 0x7f1cc8070a30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1cc8066470 msgr2=0x7f1cc8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1cc8066470 0x7f1cc8068930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc006120 tx=0x7f1ccc005b80 comp rx=0 tx=0).stop 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc158f10 msgr2=0x7f1cdc14cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 --2- 172.21.15.28:0/1844159768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cdc158f10 0x7f1cdc14cfa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1cd001a330 tx=0x7f1cd00194a0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1cdbfff640 1 -- 172.21.15.28:0/1844159768 reap_dead start 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 shutdown_connections 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 >> 172.21.15.28:0/1844159768 conn(0x7f1cdc0b0ae0 msgr2=0x7f1cdc14ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 shutdown_connections 2023-12-16T21:57:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.224+0000 7f1ce0ba6640 1 -- 172.21.15.28:0/1844159768 wait complete. 2023-12-16T21:57:49.245 INFO:teuthology.orchestra.run.smithi028.stdout:68719476743 2023-12-16T21:57:49.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-16T21:57:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 --> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae7c0674d0 con 0x7fae6006e690 2023-12-16T21:57:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae78ff9640 1 -- 172.21.15.28:0/761055090 <== osd.2 v2:172.21.15.28:6807/451272560 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae7c0674d0 con 0x7fae6006e690 2023-12-16T21:57:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 --> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8d980b6eb0 con 0x7f8d7c06e610 2023-12-16T21:57:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b57f6a640 1 -- 172.21.15.28:0/2348267755 --> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b500ccfa0 con 0x7f8b3406e570 2023-12-16T21:57:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7fae6006e690 msgr2=0x7fae60070ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7fae6006e690 0x7fae60070ab0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae600664f0 msgr2=0x7fae600689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fae600664f0 0x7fae600689b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae7400b240 tx=0x7fae7400f040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c14e2d0 msgr2=0x7fae7c149260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 --2- 172.21.15.28:0/761055090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae7c14e2d0 0x7fae7c149260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fae700ac040 tx=0x7fae7001ad40 comp rx=0 tx=0).stop 2023-12-16T21:57:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d8dffb640 1 -- 172.21.15.28:0/2457083161 <== osd.0 v2:172.21.15.28:6810/177892190 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8d980b6eb0 con 0x7f8d7c06e610 2023-12-16T21:57:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae80a19640 1 -- 172.21.15.28:0/761055090 reap_dead start 2023-12-16T21:57:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 shutdown_connections 2023-12-16T21:57:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 >> 172.21.15.28:0/761055090 conn(0x7fae7c0cfe90 msgr2=0x7fae7c152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 shutdown_connections 2023-12-16T21:57:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7fae8247c640 1 -- 172.21.15.28:0/761055090 wait complete. 2023-12-16T21:57:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b467fc640 1 -- 172.21.15.28:0/2348267755 <== osd.3 v2:172.21.15.28:6800/4021281394 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8b480a5b10 con 0x7f8b3406e570 2023-12-16T21:57:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7f8d7c06e610 msgr2=0x7f8d7c070a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7f8d7c06e610 0x7f8d7c070a30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d7c066470 msgr2=0x7f8d7c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d7c066470 0x7f8d7c068930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8d9806e980 tx=0x7f8d88009040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b0d80 msgr2=0x7f8d980c73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 --2- 172.21.15.28:0/2457083161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d980b0d80 0x7f8d980c73b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d9401a080 tx=0x7f8d9401ac90 comp rx=0 tx=0).stop 2023-12-16T21:57:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9defb640 1 -- 172.21.15.28:0/2457083161 reap_dead start 2023-12-16T21:57:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 shutdown_connections 2023-12-16T21:57:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 >> 172.21.15.28:0/2457083161 conn(0x7f8d980d0f30 msgr2=0x7f8d980999d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 -- 172.21.15.28:0/2348267755 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f8b3406e570 msgr2=0x7f8b34070990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f8b3406e570 0x7f8b34070990 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 -- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8b340664c0 msgr2=0x7f8b34068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8b340664c0 0x7f8b34068980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009fd0 tx=0x7f8b4c00b040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 -- 172.21.15.28:0/2348267755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 msgr2=0x7f8b50153d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 --2- 172.21.15.28:0/2348267755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b500cd7a0 0x7f8b50153d00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000b410 tx=0x7f8b4000b8d0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 shutdown_connections 2023-12-16T21:57:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b56507640 1 -- 172.21.15.28:0/2348267755 reap_dead start 2023-12-16T21:57:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8d9f95e640 1 -- 172.21.15.28:0/2457083161 wait complete. 2023-12-16T21:57:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 -- 172.21.15.28:0/2348267755 shutdown_connections 2023-12-16T21:57:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 -- 172.21.15.28:0/2348267755 >> 172.21.15.28:0/2348267755 conn(0x7f8b500aea50 msgr2=0x7f8b500b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 -- 172.21.15.28:0/2348267755 shutdown_connections 2023-12-16T21:57:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.232+0000 7f8b33fff640 1 -- 172.21.15.28:0/2348267755 wait complete. 2023-12-16T21:57:49.254 INFO:teuthology.orchestra.run.smithi028.stdout:68719476743 2023-12-16T21:57:49.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-16T21:57:49.255 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 --> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f72440b4110 con 0x7f723006e710 2023-12-16T21:57:49.255 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f72417fa640 1 -- 172.21.15.28:0/301676411 <== osd.4 v2:172.21.15.122:6800/1281108383 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f723c00fd40 con 0x7f723006e710 2023-12-16T21:57:49.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f723006e710 msgr2=0x7f7230070b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f723006e710 0x7f7230070b30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f72300664f0 msgr2=0x7f72300689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f72300664f0 0x7f72300689b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f72380ae000 tx=0x7f72380b5040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 msgr2=0x7f72441453e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 --2- 172.21.15.28:0/301676411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f724406e2d0 0x7f72441453e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f723400b4a0 tx=0x7f723400b960 comp rx=0 tx=0).stop 2023-12-16T21:57:49.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249195640 1 -- 172.21.15.28:0/301676411 reap_dead start 2023-12-16T21:57:49.257 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 shutdown_connections 2023-12-16T21:57:49.257 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 >> 172.21.15.28:0/301676411 conn(0x7f72440b4500 msgr2=0x7f7244150f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.257 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.244+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 shutdown_connections 2023-12-16T21:57:49.257 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.248+0000 7f7249996640 1 -- 172.21.15.28:0/301676411 wait complete. 2023-12-16T21:57:49.257 INFO:teuthology.orchestra.run.smithi028.stdout:64424509446 2023-12-16T21:57:49.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-16T21:57:49.258 INFO:teuthology.orchestra.run.smithi028.stdout:68719476742 2023-12-16T21:57:49.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-16T21:57:49.261 INFO:teuthology.orchestra.run.smithi028.stdout:60129542150 2023-12-16T21:57:49.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-16T21:57:49.263 INFO:teuthology.orchestra.run.smithi028.stdout:64424509446 2023-12-16T21:57:49.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-16T21:57:49.268 INFO:teuthology.orchestra.run.smithi028.stdout:68719476741 2023-12-16T21:57:49.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-16T21:57:49.274 INFO:teuthology.orchestra.run.smithi028.stdout:64424509448 2023-12-16T21:57:49.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-16T21:57:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.408+0000 7f361ccaf640 1 Processor -- start 2023-12-16T21:57:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.408+0000 7f361ccaf640 1 -- start start 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361ccaf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36181500a0 0x7f36180cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361ccaf640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3618159ff0 0x7f36180ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361ccaf640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361806e100 con 0x7f36180cdfb0 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361ccaf640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361815a650 con 0x7f36181500a0 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361ccaf640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36180ce980 con 0x7f3618159ff0 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f3615d9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f36180cdfb0 0x7f36181598e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51226/0 (socket says 172.21.15.28:51226) 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f3615d9b640 1 -- 172.21.15.28:0/75323034 learned_addr learned my addr 172.21.15.28:0/75323034 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361659c640 1 --2- 172.21.15.28:0/75323034 >> v2:172.21.15.28:3300/0 conn(0x7f36181500a0 0x7f36180cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361659c640 1 -- 172.21.15.28:0/75323034 >> v2:172.21.15.122:3300/0 conn(0x7f3618159ff0 msgr2=0x7f36180ce440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361659c640 1 --2- 172.21.15.28:0/75323034 >> v2:172.21.15.122:3300/0 conn(0x7f3618159ff0 0x7f36180ce440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361659c640 1 -- 172.21.15.28:0/75323034 >> v1:172.21.15.28:6789/0 conn(0x7f36180cdfb0 legacy=0x7f36181598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.412+0000 7f361659c640 1 -- 172.21.15.28:0/75323034 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36180cc740 con 0x7f36181500a0 2023-12-16T21:57:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361659c640 1 --2- 172.21.15.28:0/75323034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36181500a0 0x7f36180cd7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f360800abf0 tx=0x7f360801d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3541f995a7ca5fa4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361559a640 1 -- 172.21.15.28:0/75323034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080a69e0 con 0x7f36181500a0 2023-12-16T21:57:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361559a640 1 -- 172.21.15.28:0/75323034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36080a6b40 con 0x7f36181500a0 2023-12-16T21:57:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361559a640 1 -- 172.21.15.28:0/75323034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080ae440 con 0x7f36181500a0 2023-12-16T21:57:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361ccaf640 1 -- 172.21.15.28:0/75323034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36181500a0 msgr2=0x7f36180cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361ccaf640 1 --2- 172.21.15.28:0/75323034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36181500a0 0x7f36180cd7a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f360800abf0 tx=0x7f360801d920 comp rx=0 tx=0).stop 2023-12-16T21:57:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361ccaf640 1 -- 172.21.15.28:0/75323034 shutdown_connections 2023-12-16T21:57:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361ccaf640 1 --2- 172.21.15.28:0/75323034 >> v2:172.21.15.122:3300/0 conn(0x7f3618159ff0 0x7f36180ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361ccaf640 1 --2- 172.21.15.28:0/75323034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36181500a0 0x7f36180cd7a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361ccaf640 1 -- 172.21.15.28:0/75323034 >> 172.21.15.28:0/75323034 conn(0x7f36180aea50 msgr2=0x7f36180722c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361ccaf640 1 -- 172.21.15.28:0/75323034 shutdown_connections 2023-12-16T21:57:49.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.416+0000 7f361ccaf640 1 -- 172.21.15.28:0/75323034 wait complete. 2023-12-16T21:57:49.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f361ccaf640 1 Processor -- start 2023-12-16T21:57:49.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f361ccaf640 1 -- start start 2023-12-16T21:57:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f361ccaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36180cdfb0 0x7f361814d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f361ccaf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36181500a0 0x7f3618200940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f361ccaf640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3618159ff0 0x7f3618158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f361ccaf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361814dcc0 con 0x7f36180cdfb0 2023-12-16T21:57:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f361ccaf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3618200e80 con 0x7f36181500a0 2023-12-16T21:57:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f361ccaf640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36181559a0 con 0x7f3618159ff0 2023-12-16T21:57:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f3615d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36181500a0 0x7f3618200940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f3615d9b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36181500a0 0x7f3618200940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:37786/0 (socket says 172.21.15.28:37786) 2023-12-16T21:57:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.420+0000 7f3615d9b640 1 -- 172.21.15.28:0/3890615041 learned_addr learned my addr 172.21.15.28:0/3890615041 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f361659c640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36180cdfb0 0x7f361814d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3615d9b640 1 -- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3618159ff0 msgr2=0x7f3618158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3615d9b640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3618159ff0 0x7f3618158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3615d9b640 1 -- 172.21.15.28:0/3890615041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36180cdfb0 msgr2=0x7f361814d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3615d9b640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36180cdfb0 0x7f361814d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3615d9b640 1 -- 172.21.15.28:0/3890615041 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360800a5c0 con 0x7f36181500a0 2023-12-16T21:57:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3615d9b640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36181500a0 0x7f3618200940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f361000f850 tx=0x7f361000fd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3606ffd640 1 -- 172.21.15.28:0/3890615041 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361000d640 con 0x7f36181500a0 2023-12-16T21:57:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3606ffd640 1 -- 172.21.15.28:0/3890615041 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361000d7a0 con 0x7f36181500a0 2023-12-16T21:57:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f3606ffd640 1 -- 172.21.15.28:0/3890615041 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361000db70 con 0x7f36181500a0 2023-12-16T21:57:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe746e9e640 1 Processor -- start 2023-12-16T21:57:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe746e9e640 1 -- start start 2023-12-16T21:57:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3618157df0 con 0x7f36181500a0 2023-12-16T21:57:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36181583b0 con 0x7f36181500a0 2023-12-16T21:57:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe746e9e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe738054200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe746e9e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe73804f820 0x7fe738104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe746e9e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe738053ab0 con 0x7fe73804f3b0 2023-12-16T21:57:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe746e9e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73803aae0 con 0x7fe73804f820 2023-12-16T21:57:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe746e9e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7380537d0 con 0x7fe738100ba0 2023-12-16T21:57:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe74569b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe73804f3b0 0x7fe738104820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:58818/0 (socket says 172.21.15.28:58818) 2023-12-16T21:57:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe74569b640 1 -- 172.21.15.28:0/2050343305 learned_addr learned my addr 172.21.15.28:0/2050343305 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.424+0000 7fe745e9c640 1 --2- 172.21.15.28:0/2050343305 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe738054200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7fe74669d640 1 --2- 172.21.15.28:0/2050343305 >> v2:172.21.15.122:3300/0 conn(0x7fe73804f820 0x7fe738104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36180cdb40 con 0x7f36181500a0 2023-12-16T21:57:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7fe745e9c640 1 -- 172.21.15.28:0/2050343305 >> v2:172.21.15.122:3300/0 conn(0x7fe73804f820 msgr2=0x7fe738104f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7fe745e9c640 1 --2- 172.21.15.28:0/2050343305 >> v2:172.21.15.122:3300/0 conn(0x7fe73804f820 0x7fe738104f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7fe745e9c640 1 -- 172.21.15.28:0/2050343305 >> v1:172.21.15.122:6789/0 conn(0x7fe73804f3b0 legacy=0x7fe738104820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7fe745e9c640 1 -- 172.21.15.28:0/2050343305 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738037b00 con 0x7fe738100ba0 2023-12-16T21:57:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7f3606ffd640 1 -- 172.21.15.28:0/3890615041 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f361000dcd0 con 0x7f36181500a0 2023-12-16T21:57:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7f3606ffd640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f35f80664a0 0x7f35f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7f3606ffd640 1 -- 172.21.15.28:0/3890615041 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3610086af0 con 0x7f36181500a0 2023-12-16T21:57:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.428+0000 7f361659c640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f35f80664a0 0x7f35f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe745e9c640 1 --2- 172.21.15.28:0/2050343305 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe738054200 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe7300097c0 tx=0x7fe73001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88d06740b1163206 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe744e9a640 1 -- 172.21.15.28:0/2050343305 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7300ab650 con 0x7fe738100ba0 2023-12-16T21:57:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7f361659c640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f35f80664a0 0x7f35f8068960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36080b0000 tx=0x7f3608008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe744e9a640 1 -- 172.21.15.28:0/2050343305 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7300ab7b0 con 0x7fe738100ba0 2023-12-16T21:57:49.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe744e9a640 1 -- 172.21.15.28:0/2050343305 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7300abb40 con 0x7fe738100ba0 2023-12-16T21:57:49.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 -- 172.21.15.28:0/2050343305 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 msgr2=0x7fe738054200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 --2- 172.21.15.28:0/2050343305 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe738054200 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe7300097c0 tx=0x7fe73001c450 comp rx=0 tx=0).stop 2023-12-16T21:57:49.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 -- 172.21.15.28:0/2050343305 shutdown_connections 2023-12-16T21:57:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 --2- 172.21.15.28:0/2050343305 >> v2:172.21.15.122:3300/0 conn(0x7fe73804f820 0x7fe738104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 --2- 172.21.15.28:0/2050343305 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe738054200 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 -- 172.21.15.28:0/2050343305 >> 172.21.15.28:0/2050343305 conn(0x7fe738009f90 msgr2=0x7fe73800a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 -- 172.21.15.28:0/2050343305 shutdown_connections 2023-12-16T21:57:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7f4e07fff640 1 Processor -- start 2023-12-16T21:57:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7f4e07fff640 1 -- start start 2023-12-16T21:57:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 -- 172.21.15.28:0/2050343305 wait complete. 2023-12-16T21:57:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 Processor -- start 2023-12-16T21:57:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7f4e07fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e00054b30 0x7f4e00055330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7f4e07fff640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e000e5a70 con 0x7f4e00055870 2023-12-16T21:57:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7f4e07fff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e00051c00 con 0x7f4e000e12f0 2023-12-16T21:57:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7f4e07fff640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e00054f90 con 0x7f4e00054b30 2023-12-16T21:57:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.432+0000 7fe746e9e640 1 -- start start 2023-12-16T21:57:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d35aeb640 1 Processor -- start 2023-12-16T21:57:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d35aeb640 1 -- start start 2023-12-16T21:57:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe746e9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73804f3b0 0x7fe738049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e077fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4e00055870 0x7f4e00036680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51228/0 (socket says 172.21.15.28:51228) 2023-12-16T21:57:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e077fe640 1 -- 172.21.15.28:0/1242485343 learned_addr learned my addr 172.21.15.28:0/1242485343 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe746e9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe73804f820 0x7fe73803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe746e9e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe73819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe746e9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73804a190 con 0x7fe73804f3b0 2023-12-16T21:57:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe746e9e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73819e030 con 0x7fe73804f820 2023-12-16T21:57:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe746e9e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73819e310 con 0x7fe738100ba0 2023-12-16T21:57:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e067fc640 1 --2- 172.21.15.28:0/1242485343 >> v2:172.21.15.28:3300/0 conn(0x7f4e00054b30 0x7f4e00055330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324796594 0 0) 0x7f4e000e5a70 con 0x7f4e00055870 2023-12-16T21:57:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df0003530 con 0x7f4e00055870 2023-12-16T21:57:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39d0577640 1 Processor -- start 2023-12-16T21:57:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d35aeb640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d300cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d35aeb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7d300c51e0 0x7f7d300c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d35aeb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d300cc7e0 con 0x7f7d3015b120 2023-12-16T21:57:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d35aeb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d300cca80 con 0x7f7d300c51e0 2023-12-16T21:57:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d35aeb640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300cfdb0 con 0x7f7d300cf490 2023-12-16T21:57:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3162625490 0 0) 0x7f4df0003530 con 0x7f4e00055870 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73804f3b0 0x7fe738049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73804f3b0 0x7fe738049c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46070/0 (socket says 172.21.15.28:46070) 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 -- 172.21.15.28:0/158363261 learned_addr learned my addr 172.21.15.28:0/158363261 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe74569b640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe73804f820 0x7fe73803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe74669d640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe73819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 -- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 msgr2=0x7fe73819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe73819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 -- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe73804f820 msgr2=0x7fe73803ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe73804f820 0x7fe73803ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 -- 172.21.15.28:0/158363261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe730009420 con 0x7fe73804f3b0 2023-12-16T21:57:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe74569b640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe73804f820 0x7fe73803ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39d0577640 1 -- start start 2023-12-16T21:57:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 >> v2:172.21.15.28:3300/0 conn(0x7f4e00054b30 msgr2=0x7f4e00055330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d2ffff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d300cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d34ae9640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f7d3015b120 0x7f7d300cea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:58840/0 (socket says 172.21.15.28:58840) 2023-12-16T21:57:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d34ae9640 1 -- 172.21.15.28:0/3332191123 learned_addr learned my addr 172.21.15.28:0/3332191123 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d352ea640 1 --2- 172.21.15.28:0/3332191123 >> v2:172.21.15.122:3300/0 conn(0x7f7d300c51e0 0x7f7d300c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe745e9c640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73804f3b0 0x7fe738049c50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe73001ccf0 tx=0x7fe73001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 --2- 172.21.15.28:0/1242485343 >> v2:172.21.15.28:3300/0 conn(0x7f4e00054b30 0x7f4e00055330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 >> v1:172.21.15.122:6789/0 conn(0x7f4e000e12f0 legacy=0x7f4e000526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e00051ee0 con 0x7f4e00055870 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f4e0804ed70 con 0x7f4e00055870 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 >> v1:172.21.15.28:6789/0 conn(0x7f4e00055870 legacy=0x7f4e00036680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4df0003980 0x7f4df0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4df0004320 0x7f4df00047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4df000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e00054f90 con 0x7f4df0004fa0 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df000bf00 con 0x7f4df0004320 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df000c0c0 con 0x7f4df0003980 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d2ffff640 1 -- 172.21.15.28:0/3332191123 >> v2:172.21.15.122:3300/0 conn(0x7f7d300c51e0 msgr2=0x7f7d300c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/3332191123 >> v2:172.21.15.122:3300/0 conn(0x7f7d300c51e0 0x7f7d300c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d2ffff640 1 -- 172.21.15.28:0/3332191123 >> v1:172.21.15.122:6789/0 conn(0x7f7d3015b120 legacy=0x7f7d300cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f7d2ffff640 1 -- 172.21.15.28:0/3332191123 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300c4a00 con 0x7f7d300cf490 2023-12-16T21:57:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e077fe640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4df000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe7367fc640 1 -- 172.21.15.28:0/158363261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7300bf610 con 0x7fe73804f3b0 2023-12-16T21:57:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe7367fc640 1 -- 172.21.15.28:0/158363261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7300bf770 con 0x7fe73804f3b0 2023-12-16T21:57:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe7367fc640 1 -- 172.21.15.28:0/158363261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7300bfb00 con 0x7fe73804f3b0 2023-12-16T21:57:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e06ffd640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4df0003980 0x7f4df0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39d0577640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39c0047c20 0x7f39c00eb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39d0577640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c0037bc0 con 0x7f39c0033e40 2023-12-16T21:57:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39d0577640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c004f090 con 0x7f39c0005ea0 2023-12-16T21:57:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39d0577640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c0050210 con 0x7f39c0047c20 2023-12-16T21:57:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e077fe640 1 -- 172.21.15.28:0/1242485343 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4df0003980 msgr2=0x7f4df0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e077fe640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4df0003980 0x7f4df0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e077fe640 1 -- 172.21.15.28:0/1242485343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4df0004320 msgr2=0x7f4df00047f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:57:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e077fe640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4df0004320 0x7f4df00047f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e077fe640 1 -- 172.21.15.28:0/1242485343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e00051ee0 con 0x7f4df0004fa0 2023-12-16T21:57:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f4e067fc640 1 --2- 172.21.15.28:0/1242485343 >> v2:172.21.15.28:3300/0 conn(0x7f4e00054b30 0x7f4e00055330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:57:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe746e9e640 1 -- 172.21.15.28:0/158363261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73819e5f0 con 0x7fe73804f3b0 2023-12-16T21:57:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39ce313640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f39c0033e40 0x7f39c00eb180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51244/0 (socket says 172.21.15.28:51244) 2023-12-16T21:57:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39ce313640 1 -- 172.21.15.28:0/2394154385 learned_addr learned my addr 172.21.15.28:0/2394154385 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7fe746e9e640 1 -- 172.21.15.28:0/158363261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73819ebb0 con 0x7fe73804f3b0 2023-12-16T21:57:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.436+0000 7f39cdb12640 1 --2- 172.21.15.28:0/2394154385 >> v2:172.21.15.28:3300/0 conn(0x7f39c0047c20 0x7f39c00eb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f39cdb12640 1 -- 172.21.15.28:0/2394154385 >> v1:172.21.15.122:6789/0 conn(0x7f39c0005ea0 legacy=0x7f39c00062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f39cdb12640 1 -- 172.21.15.28:0/2394154385 >> v1:172.21.15.28:6789/0 conn(0x7f39c0033e40 legacy=0x7f39c00eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f39cdb12640 1 -- 172.21.15.28:0/2394154385 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c004fd80 con 0x7f39c0047c20 2023-12-16T21:57:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e077fe640 1 -- 172.21.15.28:0/1242485343 reap_dead start 2023-12-16T21:57:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/3332191123 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d300cf870 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009d90 tx=0x7f7d2801c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99c9baefaefb42ab server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7fe746e9e640 1 -- 172.21.15.28:0/158363261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7380089b0 con 0x7fe73804f3b0 2023-12-16T21:57:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e077fe640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4df000b990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4e08067b80 tx=0x7f4e080972c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dab01ca35599488 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7fe7367fc640 1 -- 172.21.15.28:0/158363261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe7300b5400 con 0x7fe73804f3b0 2023-12-16T21:57:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d2f7fe640 1 -- 172.21.15.28:0/3332191123 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280a59e0 con 0x7f7d300cf490 2023-12-16T21:57:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d2f7fe640 1 -- 172.21.15.28:0/3332191123 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d280a5b40 con 0x7f7d300cf490 2023-12-16T21:57:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d2f7fe640 1 -- 172.21.15.28:0/3332191123 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280ad440 con 0x7f7d300cf490 2023-12-16T21:57:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08126640 con 0x7f4df0004fa0 2023-12-16T21:57:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e08122050 con 0x7f4df0004fa0 2023-12-16T21:57:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e05ffb640 1 -- 172.21.15.28:0/1242485343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08126a40 con 0x7f4df0004fa0 2023-12-16T21:57:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3332191123 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 msgr2=0x7f7d300cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3332191123 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d300cf870 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009d90 tx=0x7f7d2801c920 comp rx=0 tx=0).stop 2023-12-16T21:57:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3332191123 shutdown_connections 2023-12-16T21:57:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3332191123 >> v2:172.21.15.122:3300/0 conn(0x7f7d300c51e0 0x7f7d300c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3332191123 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d300cf870 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3332191123 >> 172.21.15.28:0/3332191123 conn(0x7f7d300d0eb0 msgr2=0x7f7d300d12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7fe7367fc640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe724066470 0x7fe724068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3332191123 shutdown_connections 2023-12-16T21:57:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3332191123 wait complete. 2023-12-16T21:57:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e07fff640 1 -- 172.21.15.28:0/1242485343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 msgr2=0x7f4df000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e07fff640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4df000b990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4e08067b80 tx=0x7f4e080972c0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e07fff640 1 -- 172.21.15.28:0/1242485343 shutdown_connections 2023-12-16T21:57:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e07fff640 1 --2- 172.21.15.28:0/1242485343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4df000b990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e07fff640 1 -- 172.21.15.28:0/1242485343 >> 172.21.15.28:0/1242485343 conn(0x7f4e00020020 msgr2=0x7f4e00056d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7fe7367fc640 1 -- 172.21.15.28:0/158363261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe73011ea70 con 0x7fe73804f3b0 2023-12-16T21:57:49.453 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e07fff640 1 -- 172.21.15.28:0/1242485343 shutdown_connections 2023-12-16T21:57:49.453 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f4e07fff640 1 -- 172.21.15.28:0/1242485343 wait complete. 2023-12-16T21:57:49.453 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.440+0000 7f39cdb12640 1 --2- 172.21.15.28:0/2394154385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39c0047c20 0x7f39c00eb890 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f39b80097c0 tx=0x7f39b801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9b972c6356ddf4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.453 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d35aeb640 1 Processor -- start 2023-12-16T21:57:49.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d35aeb640 1 -- start start 2023-12-16T21:57:49.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e07fff640 1 Processor -- start 2023-12-16T21:57:49.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa3984640 1 Processor -- start 2023-12-16T21:57:49.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa3984640 1 -- start start 2023-12-16T21:57:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d35aeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300c51e0 0x7f7d3006cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e07fff640 1 -- start start 2023-12-16T21:57:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e07fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4e000cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e07fff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4e000cd700 0x7f4e0004de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa3984640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6a9c15b2d0 0x7f6a9c0d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa3984640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c06e410 0x7f6a9c0d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa3984640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9c06f5c0 con 0x7f6a9c15aef0 2023-12-16T21:57:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa3984640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9c06e9b0 con 0x7f6a9c15b2d0 2023-12-16T21:57:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa3984640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c06ec90 con 0x7f6a9c06e410 2023-12-16T21:57:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d35aeb640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d301534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300c51e0 0x7f7d3006cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300c51e0 0x7f7d3006cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46088/0 (socket says 172.21.15.28:46088) 2023-12-16T21:57:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 -- 172.21.15.28:0/3563726870 learned_addr learned my addr 172.21.15.28:0/3563726870 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4e000cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4e000cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46098/0 (socket says 172.21.15.28:46098) 2023-12-16T21:57:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 -- 172.21.15.28:0/733681062 learned_addr learned my addr 172.21.15.28:0/733681062 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa1720640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6a9c15aef0 0x7f6a9c0ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:58862/0 (socket says 172.21.15.28:58862) 2023-12-16T21:57:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa1720640 1 -- 172.21.15.28:0/4032905473 learned_addr learned my addr 172.21.15.28:0/4032905473 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d3015b120 0x7f7d30154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa1f21640 1 --2- 172.21.15.28:0/4032905473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c06e410 0x7f6a9c0d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa0f1f640 1 --2- 172.21.15.28:0/4032905473 >> v2:172.21.15.122:3300/0 conn(0x7f6a9c15b2d0 0x7f6a9c0d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d301534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e067fc640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4e000cd700 0x7f4e0004de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e07fff640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0004e360 0x7f4e000dd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301523d0 con 0x7f7d300c51e0 2023-12-16T21:57:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30155730 con 0x7f7d3015b120 2023-12-16T21:57:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30155170 con 0x7f7d300cf490 2023-12-16T21:57:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e07fff640 1 -- 172.21.15.28:0/733681062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e000de8c0 con 0x7f4df0004fa0 2023-12-16T21:57:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e07fff640 1 -- 172.21.15.28:0/733681062 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e00008890 con 0x7f4e0004e360 2023-12-16T21:57:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e07fff640 1 -- 172.21.15.28:0/733681062 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e00008b70 con 0x7f4e000cd700 2023-12-16T21:57:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d352ea640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d3015b120 0x7f7d30154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6a93fff640 1 -- 172.21.15.28:0/4032905473 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498917650 0 0) 0x7f6a9c06f5c0 con 0x7f6a9c15aef0 2023-12-16T21:57:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6a93fff640 1 -- 172.21.15.28:0/4032905473 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a88003530 con 0x7f6a9c15aef0 2023-12-16T21:57:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa1f21640 1 -- 172.21.15.28:0/4032905473 >> v2:172.21.15.122:3300/0 conn(0x7f6a9c15b2d0 msgr2=0x7f6a9c0d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa1f21640 1 --2- 172.21.15.28:0/4032905473 >> v2:172.21.15.122:3300/0 conn(0x7f6a9c15b2d0 0x7f6a9c0d0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa1f21640 1 -- 172.21.15.28:0/4032905473 >> v1:172.21.15.122:6789/0 conn(0x7f6a9c15aef0 legacy=0x7f6a9c0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f6aa1f21640 1 -- 172.21.15.28:0/4032905473 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c06ef70 con 0x7f6a9c06e410 2023-12-16T21:57:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f3606ffd640 1 -- 172.21.15.28:0/3890615041 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3610054090 con 0x7f36181500a0 2023-12-16T21:57:49.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e077fe640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0004e360 0x7f4e000dd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7fe74569b640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe724066470 0x7fe724068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 -- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 msgr2=0x7f7d301534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d301534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 -- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d3015b120 msgr2=0x7f7d30154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d3015b120 0x7f7d30154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 -- 172.21.15.28:0/3563726870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d28009a70 con 0x7f7d300c51e0 2023-12-16T21:57:49.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 -- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4e000cd700 msgr2=0x7f4e0004de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4e000cd700 0x7f4e0004de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 -- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0004e360 msgr2=0x7f4e000dd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0004e360 0x7f4e000dd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 -- 172.21.15.28:0/733681062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e00008e50 con 0x7f4df0004fa0 2023-12-16T21:57:49.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d301534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f7d34ae9640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300c51e0 0x7f7d3006cc30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400c400 tx=0x7f7d2400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.444+0000 7f4e06ffd640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4e000cd1c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4df800e440 tx=0x7f4df800e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f39cd311640 1 -- 172.21.15.28:0/2394154385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b80ab600 con 0x7f39c0047c20 2023-12-16T21:57:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f39cd311640 1 -- 172.21.15.28:0/2394154385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b80a7050 con 0x7f39c0047c20 2023-12-16T21:57:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f39cd311640 1 -- 172.21.15.28:0/2394154385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b80ab980 con 0x7f39c0047c20 2023-12-16T21:57:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4def7fe640 1 -- 172.21.15.28:0/733681062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df80052e0 con 0x7f4df0004fa0 2023-12-16T21:57:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4e07fff640 1 -- 172.21.15.28:0/733681062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e00009130 con 0x7f4df0004fa0 2023-12-16T21:57:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4e07fff640 1 -- 172.21.15.28:0/733681062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e00008440 con 0x7f4df0004fa0 2023-12-16T21:57:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4def7fe640 1 -- 172.21.15.28:0/733681062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df8005440 con 0x7f4df0004fa0 2023-12-16T21:57:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4def7fe640 1 -- 172.21.15.28:0/733681062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df8003820 con 0x7f4df0004fa0 2023-12-16T21:57:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7fe74569b640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe724066470 0x7fe724068930 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe740069330 tx=0x7fe74006d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4e07fff640 1 -- 172.21.15.28:0/733681062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e00055cc0 con 0x7f4df0004fa0 2023-12-16T21:57:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7fe7367fc640 1 -- 172.21.15.28:0/158363261 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe7300ebf80 con 0x7fe73804f3b0 2023-12-16T21:57:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f6aa1f21640 1 --2- 172.21.15.28:0/4032905473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c06e410 0x7f6a9c0d10e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009d90 tx=0x7f6a9401c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9b338aa43d120fe server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4def7fe640 1 -- 172.21.15.28:0/733681062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4df8003980 con 0x7f4df0004fa0 2023-12-16T21:57:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4def7fe640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4df4066470 0x7f4df4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4def7fe640 1 -- 172.21.15.28:0/733681062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4df8084350 con 0x7f4df0004fa0 2023-12-16T21:57:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.448+0000 7f4e067fc640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4df4066470 0x7f4df4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f4e067fc640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4df4066470 0x7f4df4068930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e0004aaa0 tx=0x7f4dfc009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f4def7fe640 1 -- 172.21.15.28:0/733681062 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4df80518f0 con 0x7f4df0004fa0 2023-12-16T21:57:49.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6a93fff640 1 -- 172.21.15.28:0/4032905473 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a940a58f0 con 0x7f6a9c06e410 2023-12-16T21:57:49.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6a93fff640 1 -- 172.21.15.28:0/4032905473 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a940a5a50 con 0x7f6a9c06e410 2023-12-16T21:57:49.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6a93fff640 1 -- 172.21.15.28:0/4032905473 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a940a5de0 con 0x7f6a9c06e410 2023-12-16T21:57:49.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 -- 172.21.15.28:0/4032905473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c06e410 msgr2=0x7f6a9c0d10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 --2- 172.21.15.28:0/4032905473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c06e410 0x7f6a9c0d10e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009d90 tx=0x7f6a9401c920 comp rx=0 tx=0).stop 2023-12-16T21:57:49.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 -- 172.21.15.28:0/4032905473 shutdown_connections 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 --2- 172.21.15.28:0/4032905473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c06e410 0x7f6a9c0d10e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 --2- 172.21.15.28:0/4032905473 >> v2:172.21.15.122:3300/0 conn(0x7f6a9c15b2d0 0x7f6a9c0d0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 -- 172.21.15.28:0/4032905473 >> 172.21.15.28:0/4032905473 conn(0x7f6a9c0999d0 msgr2=0x7f6a9c099de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 -- 172.21.15.28:0/4032905473 shutdown_connections 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 -- 172.21.15.28:0/4032905473 wait complete. 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 Processor -- start 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.452+0000 7f6aa3984640 1 -- start start 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa3984640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a9c06e410 0x7f6a9c0c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a9c06e410 0x7f6a9c0c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a9c06e410 0x7f6a9c0c7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46104/0 (socket says 172.21.15.28:46104) 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa3984640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c15aef0 0x7f6a9c0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa3984640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9c15b2d0 0x7f6a9c14b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa3984640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c0b16c0 con 0x7f6a9c06e410 2023-12-16T21:57:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa3984640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c0c90d0 con 0x7f6a9c15b2d0 2023-12-16T21:57:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa3984640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c14ae80 con 0x7f6a9c15aef0 2023-12-16T21:57:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 -- 172.21.15.28:0/2139593567 learned_addr learned my addr 172.21.15.28:0/2139593567 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f39d0577640 1 -- 172.21.15.28:0/2394154385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39c0047c20 msgr2=0x7f39c00eb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f39d0577640 1 --2- 172.21.15.28:0/2394154385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39c0047c20 0x7f39c00eb890 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f39b80097c0 tx=0x7f39b801c3a0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1f21640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9c15b2d0 0x7f6a9c14b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 -- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c15aef0 msgr2=0x7f6a9c0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c15aef0 0x7f6a9c0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 -- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9c15b2d0 msgr2=0x7f6a9c14b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9c15b2d0 0x7f6a9c14b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 -- 172.21.15.28:0/2139593567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a94009a70 con 0x7f6a9c06e410 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1f21640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9c15b2d0 0x7f6a9c14b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa1720640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a9c06e410 0x7f6a9c0c7ee0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6a9800c500 tx=0x7f6a9800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6a91ffb640 1 -- 172.21.15.28:0/2139593567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a98005cf0 con 0x7f6a9c06e410 2023-12-16T21:57:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6a91ffb640 1 -- 172.21.15.28:0/2139593567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a98005e50 con 0x7f6a9c06e410 2023-12-16T21:57:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6a91ffb640 1 -- 172.21.15.28:0/2139593567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a98007310 con 0x7f6a9c06e410 2023-12-16T21:57:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa3984640 1 -- 172.21.15.28:0/2139593567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9c0c84c0 con 0x7f6a9c06e410 2023-12-16T21:57:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6aa3984640 1 -- 172.21.15.28:0/2139593567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9c0c8a80 con 0x7f6a9c06e410 2023-12-16T21:57:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f39d0577640 1 -- 172.21.15.28:0/2394154385 shutdown_connections 2023-12-16T21:57:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f39d0577640 1 --2- 172.21.15.28:0/2394154385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39c0047c20 0x7f39c00eb890 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f39d0577640 1 -- 172.21.15.28:0/2394154385 >> 172.21.15.28:0/2394154385 conn(0x7f39c0009fc0 msgr2=0x7f39c000a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f39d0577640 1 -- 172.21.15.28:0/2394154385 shutdown_connections 2023-12-16T21:57:49.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/3563726870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24015e90 con 0x7f7d300c51e0 2023-12-16T21:57:49.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/3563726870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d240058c0 con 0x7f7d300c51e0 2023-12-16T21:57:49.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d30155450 con 0x7f7d300c51e0 2023-12-16T21:57:49.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d30152fc0 con 0x7f7d300c51e0 2023-12-16T21:57:49.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f6a7f7fe640 1 -- 172.21.15.28:0/2139593567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a78005260 con 0x7f6a9c06e410 2023-12-16T21:57:49.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.456+0000 7f39d0577640 1 -- 172.21.15.28:0/2394154385 wait complete. 2023-12-16T21:57:49.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39d0577640 1 Processor -- start 2023-12-16T21:57:49.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39d0577640 1 -- start start 2023-12-16T21:57:49.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39d0577640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39c0005ea0 0x7f39c00dc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39d0577640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f39c0033e40 0x7f39c004e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39d0577640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39c0047c20 0x7f39c0034510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39d0577640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c004d340 con 0x7f39c0005ea0 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39d0577640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c004c6c0 con 0x7f39c0047c20 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39d0577640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c004c9a0 con 0x7f39c0033e40 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f39c0033e40 0x7f39c004e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f39c0033e40 0x7f39c004e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:60102/0 (socket says 172.21.15.28:60102) 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 -- 172.21.15.28:0/1994893552 learned_addr learned my addr 172.21.15.28:0/1994893552 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39ceb14640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39c0047c20 0x7f39c0034510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f6a91ffb640 1 -- 172.21.15.28:0/2139593567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6a98007470 con 0x7f6a9c06e410 2023-12-16T21:57:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f6a91ffb640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f6a91ffb640 1 -- 172.21.15.28:0/2139593567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a98083950 con 0x7f6a9c06e410 2023-12-16T21:57:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f6aa0f1f640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 -- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39c0047c20 msgr2=0x7f39c0034510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39c0047c20 0x7f39c0034510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 -- 172.21.15.28:0/1994893552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39c0005ea0 msgr2=0x7f39c00dc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39c0005ea0 0x7f39c00dc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 -- 172.21.15.28:0/1994893552 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b8009420 con 0x7f39c0033e40 2023-12-16T21:57:49.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f39cdb12640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f39c0033e40 0x7f39c004e510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f39b80097c0 tx=0x7f39b8019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/3563726870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24005c90 con 0x7f7d300c51e0 2023-12-16T21:57:49.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.460+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/3563726870 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7d2401b020 con 0x7f7d300c51e0 2023-12-16T21:57:49.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f6aa0f1f640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c006aa0 tx=0x7f6a8c006a30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f7d2d7fa640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7d1c0665c0 0x7f7d1c068a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/3563726870 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d2408b900 con 0x7f7d300c51e0 2023-12-16T21:57:49.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7d1c0665c0 0x7f7d1c068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f39beffd640 1 -- 172.21.15.28:0/1994893552 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b80ba070 con 0x7f39c0033e40 2023-12-16T21:57:49.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d10005260 con 0x7f7d300c51e0 2023-12-16T21:57:49.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f39beffd640 1 -- 172.21.15.28:0/1994893552 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b80b4610 con 0x7f39c0033e40 2023-12-16T21:57:49.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f39d0577640 1 -- 172.21.15.28:0/1994893552 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c004cc80 con 0x7f39c0033e40 2023-12-16T21:57:49.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f39beffd640 1 -- 172.21.15.28:0/1994893552 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b80b49a0 con 0x7f39c0033e40 2023-12-16T21:57:49.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f39d0577640 1 -- 172.21.15.28:0/1994893552 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c00dc830 con 0x7f39c0033e40 2023-12-16T21:57:49.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f7d2ffff640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7d1c0665c0 0x7f7d1c068a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d280af000 tx=0x7f7d280a7070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.464+0000 7f39bcff9640 1 -- 172.21.15.28:0/1994893552 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a4005260 con 0x7f39c0033e40 2023-12-16T21:57:49.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.468+0000 7f39beffd640 1 -- 172.21.15.28:0/1994893552 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f39b80b2040 con 0x7f39c0033e40 2023-12-16T21:57:49.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.468+0000 7f39beffd640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f39ac0664c0 0x7f39ac068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.468+0000 7f39beffd640 1 -- 172.21.15.28:0/1994893552 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39b811e620 con 0x7f39c0033e40 2023-12-16T21:57:49.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.468+0000 7f6a91ffb640 1 -- 172.21.15.28:0/2139593567 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6a98050e40 con 0x7f6a9c06e410 2023-12-16T21:57:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.468+0000 7f39ce313640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f39ac0664c0 0x7f39ac068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.472+0000 7f39ce313640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f39ac0664c0 0x7f39ac068980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f39c4009fd0 tx=0x7f39c4001c60 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.476+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/3563726870 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7d24058e10 con 0x7f7d300c51e0 2023-12-16T21:57:49.486 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.480+0000 7f39beffd640 1 -- 172.21.15.28:0/1994893552 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f39b80eba80 con 0x7f39c0033e40 2023-12-16T21:57:49.506 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.504+0000 7f42d8622640 1 Processor -- start 2023-12-16T21:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.504+0000 7f42d8622640 1 -- start start 2023-12-16T21:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d8622640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d015cdb0 0x7f42d00cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d8622640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42d015d190 0x7f42d00b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d8622640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d00ce840 con 0x7f42d0159490 2023-12-16T21:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d8622640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d00b2500 con 0x7f42d015d190 2023-12-16T21:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d8622640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d015dd00 con 0x7f42d015cdb0 2023-12-16T21:57:49.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f0665366640 1 Processor -- start 2023-12-16T21:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d63be640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d015cdb0 0x7f42d00cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d63be640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d015cdb0 0x7f42d00cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:60116/0 (socket says 172.21.15.28:60116) 2023-12-16T21:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d63be640 1 -- 172.21.15.28:0/4025624362 learned_addr learned my addr 172.21.15.28:0/4025624362 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d5bbd640 1 --2- 172.21.15.28:0/4025624362 >> v2:172.21.15.28:3300/0 conn(0x7f42d015d190 0x7f42d00b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f0665366640 1 -- start start 2023-12-16T21:57:49.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f0665366640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06600b2fd0 0x7f06600b4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f0665366640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f06600b4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f0665366640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06600b39b0 con 0x7f066014b250 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f0665366640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06600b3f70 con 0x7f06600b2fd0 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f0665366640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06600b3c90 con 0x7f06600b3450 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f065e7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06600b2fd0 0x7f06600b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f065e7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f06600b2fd0 0x7f06600b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:37872/0 (socket says 172.21.15.28:37872) 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f065e7fc640 1 -- 172.21.15.28:0/1359446563 learned_addr learned my addr 172.21.15.28:0/1359446563 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f065f7fe640 1 --2- 172.21.15.28:0/1359446563 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f06600b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d63be640 1 -- 172.21.15.28:0/4025624362 >> v2:172.21.15.28:3300/0 conn(0x7f42d015d190 msgr2=0x7f42d00b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d63be640 1 --2- 172.21.15.28:0/4025624362 >> v2:172.21.15.28:3300/0 conn(0x7f42d015d190 0x7f42d00b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d63be640 1 -- 172.21.15.28:0/4025624362 >> v1:172.21.15.122:6789/0 conn(0x7f42d0159490 legacy=0x7f42d015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.508+0000 7f42d63be640 1 -- 172.21.15.28:0/4025624362 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d00ce560 con 0x7f42d015cdb0 2023-12-16T21:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f065f7fe640 1 -- 172.21.15.28:0/1359446563 >> v2:172.21.15.122:3300/0 conn(0x7f06600b2fd0 msgr2=0x7f06600b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f065f7fe640 1 --2- 172.21.15.28:0/1359446563 >> v2:172.21.15.122:3300/0 conn(0x7f06600b2fd0 0x7f06600b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f065f7fe640 1 -- 172.21.15.28:0/1359446563 >> v1:172.21.15.122:6789/0 conn(0x7f066014b250 legacy=0x7f06600cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f065f7fe640 1 -- 172.21.15.28:0/1359446563 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0660071750 con 0x7f06600b3450 2023-12-16T21:57:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d63be640 1 --2- 172.21.15.28:0/4025624362 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d015cdb0 0x7f42d00cb690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009870 tx=0x7f42c001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d15fb6a0803404f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d53bc640 1 -- 172.21.15.28:0/4025624362 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c00a7070 con 0x7f42d015cdb0 2023-12-16T21:57:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d53bc640 1 -- 172.21.15.28:0/4025624362 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c00ab430 con 0x7f42d015cdb0 2023-12-16T21:57:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d53bc640 1 -- 172.21.15.28:0/4025624362 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c00ab7c0 con 0x7f42d015cdb0 2023-12-16T21:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d8622640 1 -- 172.21.15.28:0/4025624362 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d015cdb0 msgr2=0x7f42d00cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d8622640 1 --2- 172.21.15.28:0/4025624362 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d015cdb0 0x7f42d00cb690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009870 tx=0x7f42c001c470 comp rx=0 tx=0).stop 2023-12-16T21:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d8622640 1 -- 172.21.15.28:0/4025624362 shutdown_connections 2023-12-16T21:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d8622640 1 --2- 172.21.15.28:0/4025624362 >> v2:172.21.15.28:3300/0 conn(0x7f42d015d190 0x7f42d00b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d8622640 1 --2- 172.21.15.28:0/4025624362 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d015cdb0 0x7f42d00cb690 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d8622640 1 -- 172.21.15.28:0/4025624362 >> 172.21.15.28:0/4025624362 conn(0x7f42d006c2c0 msgr2=0x7f42d006c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d8622640 1 -- 172.21.15.28:0/4025624362 shutdown_connections 2023-12-16T21:57:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f42d8622640 1 -- 172.21.15.28:0/4025624362 wait complete. 2023-12-16T21:57:49.515 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.512+0000 7f065f7fe640 1 --2- 172.21.15.28:0/1359446563 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f06600b4950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0658009d90 tx=0x7f065801c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64da30f04e1ea511 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.515 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d8622640 1 Processor -- start 2023-12-16T21:57:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f065dffb640 1 -- 172.21.15.28:0/1359446563 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580a5990 con 0x7f06600b3450 2023-12-16T21:57:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f065dffb640 1 -- 172.21.15.28:0/1359446563 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06580a5af0 con 0x7f06600b3450 2023-12-16T21:57:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f065dffb640 1 -- 172.21.15.28:0/1359446563 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580ad440 con 0x7f06600b3450 2023-12-16T21:57:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f0665366640 1 -- 172.21.15.28:0/1359446563 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 msgr2=0x7f06600b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f0665366640 1 --2- 172.21.15.28:0/1359446563 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f06600b4950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0658009d90 tx=0x7f065801c8d0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d8622640 1 -- start start 2023-12-16T21:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d8622640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d0159490 0x7f42d01f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d8622640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d015cdb0 0x7f42d01471f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d015cdb0 0x7f42d01471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d015cdb0 0x7f42d01471f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46136/0 (socket says 172.21.15.28:46136) 2023-12-16T21:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 -- 172.21.15.28:0/4285303449 learned_addr learned my addr 172.21.15.28:0/4285303449 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d8622640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42d015d190 0x7f42d00aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d63be640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d0159490 0x7f42d01f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d01f25c0 con 0x7f42d015cdb0 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d014eb20 con 0x7f42d015d190 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d014ee00 con 0x7f42d0159490 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d6bbf640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42d015d190 0x7f42d00aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 -- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d0159490 msgr2=0x7f42d01f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d0159490 0x7f42d01f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 -- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42d015d190 msgr2=0x7f42d00aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42d015d190 0x7f42d00aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 -- 172.21.15.28:0/4285303449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c00094d0 con 0x7f42d015cdb0 2023-12-16T21:57:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.516+0000 7f42d5bbd640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d015cdb0 0x7f42d01471f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f42c8012f40 tx=0x7f42c8010550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 -- 172.21.15.28:0/1359446563 shutdown_connections 2023-12-16T21:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 --2- 172.21.15.28:0/1359446563 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f06600b4950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 --2- 172.21.15.28:0/1359446563 >> v2:172.21.15.122:3300/0 conn(0x7f06600b2fd0 0x7f06600b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 -- 172.21.15.28:0/1359446563 >> 172.21.15.28:0/1359446563 conn(0x7f066006d530 msgr2=0x7f066006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 -- 172.21.15.28:0/1359446563 shutdown_connections 2023-12-16T21:57:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 -- 172.21.15.28:0/1359446563 wait complete. 2023-12-16T21:57:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 Processor -- start 2023-12-16T21:57:49.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 -- start start 2023-12-16T21:57:49.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06600b2fd0 0x7f066013b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f0660163e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f066014b250 0x7f0660158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066013aeb0 con 0x7f06600b2fd0 2023-12-16T21:57:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0660164780 con 0x7f066014b250 2023-12-16T21:57:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f0665366640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066013b150 con 0x7f06600b3450 2023-12-16T21:57:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f42c6ffd640 1 -- 172.21.15.28:0/4285303449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8018680 con 0x7f42d015cdb0 2023-12-16T21:57:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d014f0e0 con 0x7f42d015cdb0 2023-12-16T21:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f065effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06600b2fd0 0x7f066013b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f065effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06600b2fd0 0x7f066013b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46144/0 (socket says 172.21.15.28:46144) 2023-12-16T21:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f065effd640 1 -- 172.21.15.28:0/1999064608 learned_addr learned my addr 172.21.15.28:0/1999064608 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f065e7fc640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f0660163e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f065f7fe640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f066014b250 0x7f0660158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.520+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d014f6a0 con 0x7f42d015cdb0 2023-12-16T21:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f065effd640 1 -- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 msgr2=0x7f0660163e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f065effd640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f0660163e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f065effd640 1 -- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f066014b250 msgr2=0x7f0660158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f065effd640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f066014b250 0x7f0660158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f065effd640 1 -- 172.21.15.28:0/1999064608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0658009a70 con 0x7f06600b2fd0 2023-12-16T21:57:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f065effd640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06600b2fd0 0x7f066013b950 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f065400c3c0 tx=0x7f065400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f42c6ffd640 1 -- 172.21.15.28:0/4285303449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c8013040 con 0x7f42d015cdb0 2023-12-16T21:57:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f42c6ffd640 1 -- 172.21.15.28:0/4285303449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8018a00 con 0x7f42d015cdb0 2023-12-16T21:57:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f42c6ffd640 1 -- 172.21.15.28:0/4285303449 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f42c8018b60 con 0x7f42d015cdb0 2023-12-16T21:57:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f42c6ffd640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42bc066750 0x7f42bc068c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f42c6ffd640 1 -- 172.21.15.28:0/4285303449 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42c8005e40 con 0x7f42d015cdb0 2023-12-16T21:57:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.524+0000 7f42d63be640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42bc066750 0x7f42bc068c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f42d63be640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42bc066750 0x7f42bc068c10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42c00af000 tx=0x7f42c0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f06477fe640 1 -- 172.21.15.28:0/1999064608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540052d0 con 0x7f06600b2fd0 2023-12-16T21:57:49.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f06477fe640 1 -- 172.21.15.28:0/1999064608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654005430 con 0x7f06600b2fd0 2023-12-16T21:57:49.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066013b430 con 0x7f06600b2fd0 2023-12-16T21:57:49.528 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f06477fe640 1 -- 172.21.15.28:0/1999064608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654003840 con 0x7f06600b2fd0 2023-12-16T21:57:49.528 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0660164350 con 0x7f06600b2fd0 2023-12-16T21:57:49.528 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b8005260 con 0x7f42d015cdb0 2023-12-16T21:57:49.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f06477fe640 1 -- 172.21.15.28:0/1999064608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f06540039a0 con 0x7f06600b2fd0 2023-12-16T21:57:49.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f06477fe640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f064c0664f0 0x7f064c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:49.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066013b670 con 0x7f06600b2fd0 2023-12-16T21:57:49.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.528+0000 7f065e7fc640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f064c0664f0 0x7f064c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:49.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.532+0000 7f065e7fc640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f064c0664f0 0x7f064c0689b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f06600af230 tx=0x7f0650008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:49.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.536+0000 7f42c6ffd640 1 -- 172.21.15.28:0/4285303449 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f42c8055f70 con 0x7f42d015cdb0 2023-12-16T21:57:49.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.540+0000 7f06477fe640 1 -- 172.21.15.28:0/1999064608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0654054210 con 0x7f06600b2fd0 2023-12-16T21:57:49.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.548+0000 7f06477fe640 1 -- 172.21.15.28:0/1999064608 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f066013b670 con 0x7f06600b2fd0 2023-12-16T21:57:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.740+0000 7f4e07fff640 1 -- 172.21.15.28:0/733681062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4e00055cc0 con 0x7f4df0004fa0 2023-12-16T21:57:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.740+0000 7f4def7fe640 1 -- 172.21.15.28:0/733681062 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e00055cc0 con 0x7f4df0004fa0 2023-12-16T21:57:49.745 INFO:teuthology.orchestra.run.smithi028.stdout:64424509446 2023-12-16T21:57:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 -- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4df4066470 msgr2=0x7f4df4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4df4066470 0x7f4df4068930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e0004aaa0 tx=0x7f4dfc009190 comp rx=0 tx=0).stop 2023-12-16T21:57:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 -- 172.21.15.28:0/733681062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 msgr2=0x7f4e000cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4e000cd1c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4df800e440 tx=0x7f4df800e900 comp rx=0 tx=0).stop 2023-12-16T21:57:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 -- 172.21.15.28:0/733681062 shutdown_connections 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4df4066470 0x7f4df4068930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e0004e360 0x7f4e000dd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4e000cd700 0x7f4e0004de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 --2- 172.21.15.28:0/733681062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0004fa0 0x7f4e000cd1c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 -- 172.21.15.28:0/733681062 >> 172.21.15.28:0/733681062 conn(0x7f4e00020020 msgr2=0x7f4e00004f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 -- 172.21.15.28:0/733681062 shutdown_connections 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7f4ded7fa640 1 -- 172.21.15.28:0/733681062 wait complete. 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.744+0000 7fe746e9e640 1 -- 172.21.15.28:0/158363261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe738005370 con 0x7fe73804f3b0 2023-12-16T21:57:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.748+0000 7fe7367fc640 1 -- 172.21.15.28:0/158363261 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe7300ef5a0 con 0x7fe73804f3b0 2023-12-16T21:57:49.750 INFO:teuthology.orchestra.run.smithi028.stdout:64424509446 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 -- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe724066470 msgr2=0x7fe724068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe724066470 0x7fe724068930 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe740069330 tx=0x7fe74006d040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 -- 172.21.15.28:0/158363261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73804f3b0 msgr2=0x7fe738049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73804f3b0 0x7fe738049c50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe73001ccf0 tx=0x7fe73001ac90 comp rx=0 tx=0).stop 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 -- 172.21.15.28:0/158363261 shutdown_connections 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe724066470 0x7fe724068930 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe738100ba0 0x7fe73819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe73804f820 0x7fe73803ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 --2- 172.21.15.28:0/158363261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe73804f3b0 0x7fe738049c50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 -- 172.21.15.28:0/158363261 >> 172.21.15.28:0/158363261 conn(0x7fe738009f90 msgr2=0x7fe738038f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 -- 172.21.15.28:0/158363261 shutdown_connections 2023-12-16T21:57:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.756+0000 7fe723fff640 1 -- 172.21.15.28:0/158363261 wait complete. 2023-12-16T21:57:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.764+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f36180693d0 con 0x7f36181500a0 2023-12-16T21:57:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.764+0000 7f3606ffd640 1 -- 172.21.15.28:0/3890615041 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f361001a020 con 0x7f36181500a0 2023-12-16T21:57:49.767 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.1 2023-12-16T21:57:49.767 INFO:teuthology.orchestra.run.smithi028.stdout:68719476742 2023-12-16T21:57:49.767 DEBUG:teuthology.parallel:result is None 2023-12-16T21:57:49.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.768+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f35f80664a0 msgr2=0x7f35f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.768+0000 7f361ccaf640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f35f80664a0 0x7f35f8068960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36080b0000 tx=0x7f3608008040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.768+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36181500a0 msgr2=0x7f3618200940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.768+0000 7f361ccaf640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36181500a0 0x7f3618200940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f361000f850 tx=0x7f361000fd10 comp rx=0 tx=0).stop 2023-12-16T21:57:49.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.768+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 shutdown_connections 2023-12-16T21:57:49.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.768+0000 7f361ccaf640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f35f80664a0 0x7f35f8068960 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.768+0000 7f361ccaf640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3618159ff0 0x7f3618158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.772+0000 7f361ccaf640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f36181500a0 0x7f3618200940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.772+0000 7f361ccaf640 1 --2- 172.21.15.28:0/3890615041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36180cdfb0 0x7f361814d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.772+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 >> 172.21.15.28:0/3890615041 conn(0x7f36180aea50 msgr2=0x7f36180721a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.772+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 shutdown_connections 2023-12-16T21:57:49.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.772+0000 7f361ccaf640 1 -- 172.21.15.28:0/3890615041 wait complete. 2023-12-16T21:57:49.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.0 2023-12-16T21:57:49.777 DEBUG:teuthology.parallel:result is None 2023-12-16T21:57:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 -- 172.21.15.28:0/2139593567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6a780057e0 con 0x7f6a9c06e410 2023-12-16T21:57:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a91ffb640 1 -- 172.21.15.28:0/2139593567 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a98017090 con 0x7f6a9c06e410 2023-12-16T21:57:49.792 INFO:teuthology.orchestra.run.smithi028.stdout:60129542150 2023-12-16T21:57:49.793 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.6 2023-12-16T21:57:49.793 DEBUG:teuthology.parallel:result is None 2023-12-16T21:57:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 -- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 msgr2=0x7f6a80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c006aa0 tx=0x7f6a8c006a30 comp rx=0 tx=0).stop 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 -- 172.21.15.28:0/2139593567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a9c06e410 msgr2=0x7f6a9c0c7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a9c06e410 0x7f6a9c0c7ee0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6a9800c500 tx=0x7f6a9800c9c0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 -- 172.21.15.28:0/2139593567 shutdown_connections 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6a800664a0 0x7f6a80068960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a9c15b2d0 0x7f6a9c14b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6a9c15aef0 0x7f6a9c0b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 --2- 172.21.15.28:0/2139593567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a9c06e410 0x7f6a9c0c7ee0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 -- 172.21.15.28:0/2139593567 >> 172.21.15.28:0/2139593567 conn(0x7f6a9c0999d0 msgr2=0x7f6a9c172450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 -- 172.21.15.28:0/2139593567 shutdown_connections 2023-12-16T21:57:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.792+0000 7f6a7f7fe640 1 -- 172.21.15.28:0/2139593567 wait complete. 2023-12-16T21:57:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7d10005510 con 0x7f7d300c51e0 2023-12-16T21:57:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d2d7fa640 1 -- 172.21.15.28:0/3563726870 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d24023030 con 0x7f7d300c51e0 2023-12-16T21:57:49.804 INFO:teuthology.orchestra.run.smithi028.stdout:68719476743 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7d1c0665c0 msgr2=0x7f7d1c068a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7d1c0665c0 0x7f7d1c068a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d280af000 tx=0x7f7d280a7070 comp rx=0 tx=0).stop 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300c51e0 msgr2=0x7f7d3006cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300c51e0 0x7f7d3006cc30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400c400 tx=0x7f7d2400c8c0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 shutdown_connections 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7d1c0665c0 0x7f7d1c068a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d3015b120 0x7f7d30154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d300cf490 0x7f7d301534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 --2- 172.21.15.28:0/3563726870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d300c51e0 0x7f7d3006cc30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 >> 172.21.15.28:0/3563726870 conn(0x7f7d300d0eb0 msgr2=0x7f7d3006bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 shutdown_connections 2023-12-16T21:57:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.804+0000 7f7d35aeb640 1 -- 172.21.15.28:0/3563726870 wait complete. 2023-12-16T21:57:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.808+0000 7f39bcff9640 1 -- 172.21.15.28:0/1994893552 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f39a40057e0 con 0x7f39c0033e40 2023-12-16T21:57:49.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2023-12-16T21:57:49.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.808+0000 7f39beffd640 1 -- 172.21.15.28:0/1994893552 <== mon.2 v2:172.21.15.122:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f39b80b0090 con 0x7f39c0033e40 2023-12-16T21:57:49.811 INFO:teuthology.orchestra.run.smithi028.stdout:68719476743 2023-12-16T21:57:49.811 DEBUG:teuthology.parallel:result is None 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 -- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f39ac0664c0 msgr2=0x7f39ac068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f39ac0664c0 0x7f39ac068980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f39c4009fd0 tx=0x7f39c4001c60 comp rx=0 tx=0).stop 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 -- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f39c0033e40 msgr2=0x7f39c004e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f39c0033e40 0x7f39c004e510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f39b80097c0 tx=0x7f39b8019140 comp rx=0 tx=0).stop 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 -- 172.21.15.28:0/1994893552 shutdown_connections 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f39ac0664c0 0x7f39ac068980 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39c0047c20 0x7f39c0034510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f39c0033e40 0x7f39c004e510 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 --2- 172.21.15.28:0/1994893552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39c0005ea0 0x7f39c00dc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 -- 172.21.15.28:0/1994893552 >> 172.21.15.28:0/1994893552 conn(0x7f39c0009fc0 msgr2=0x7f39c00e4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 -- 172.21.15.28:0/1994893552 shutdown_connections 2023-12-16T21:57:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.812+0000 7f39bcff9640 1 -- 172.21.15.28:0/1994893552 wait complete. 2023-12-16T21:57:49.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.7 2023-12-16T21:57:49.824 DEBUG:teuthology.parallel:result is None 2023-12-16T21:57:49.830 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.5 2023-12-16T21:57:49.830 DEBUG:teuthology.parallel:result is None 2023-12-16T21:57:49.832 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f42b8005510 con 0x7f42d015cdb0 2023-12-16T21:57:49.832 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42c6ffd640 1 -- 172.21.15.28:0/4285303449 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f42c8059590 con 0x7f42d015cdb0 2023-12-16T21:57:49.832 INFO:teuthology.orchestra.run.smithi028.stdout:64424509448 2023-12-16T21:57:49.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42bc066750 msgr2=0x7f42bc068c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42bc066750 0x7f42bc068c10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42c00af000 tx=0x7f42c0008040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d015cdb0 msgr2=0x7f42d01471f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d015cdb0 0x7f42d01471f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f42c8012f40 tx=0x7f42c8010550 comp rx=0 tx=0).stop 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 shutdown_connections 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f42bc066750 0x7f42bc068c10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42d015d190 0x7f42d00aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d015cdb0 0x7f42d01471f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 --2- 172.21.15.28:0/4285303449 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f42d0159490 0x7f42d01f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 >> 172.21.15.28:0/4285303449 conn(0x7f42d006c2c0 msgr2=0x7f42d00cb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 shutdown_connections 2023-12-16T21:57:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.832+0000 7f42d8622640 1 -- 172.21.15.28:0/4285303449 wait complete. 2023-12-16T21:57:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f066013b670 con 0x7f06600b2fd0 2023-12-16T21:57:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f06477fe640 1 -- 172.21.15.28:0/1999064608 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f066013b670 con 0x7f06600b2fd0 2023-12-16T21:57:49.840 INFO:teuthology.orchestra.run.smithi028.stdout:68719476741 2023-12-16T21:57:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f064c0664f0 msgr2=0x7f064c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f064c0664f0 0x7f064c0689b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f06600af230 tx=0x7f0650008040 comp rx=0 tx=0).stop 2023-12-16T21:57:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06600b2fd0 msgr2=0x7f066013b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06600b2fd0 0x7f066013b950 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f065400c3c0 tx=0x7f065400c880 comp rx=0 tx=0).stop 2023-12-16T21:57:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 shutdown_connections 2023-12-16T21:57:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f064c0664f0 0x7f064c0689b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f066014b250 0x7f0660158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f06600b3450 0x7f0660163e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 --2- 172.21.15.28:0/1999064608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06600b2fd0 0x7f066013b950 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 >> 172.21.15.28:0/1999064608 conn(0x7f066006d530 msgr2=0x7f066006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 shutdown_connections 2023-12-16T21:57:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:49.840+0000 7f0665366640 1 -- 172.21.15.28:0/1999064608 wait complete. 2023-12-16T21:57:49.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.3 2023-12-16T21:57:49.851 DEBUG:teuthology.parallel:result is None 2023-12-16T21:57:49.858 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476741 got 68719476741 for osd.4 2023-12-16T21:57:49.858 DEBUG:teuthology.parallel:result is None 2023-12-16T21:57:49.858 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-16T21:57:49.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:57:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c23774640 1 Processor -- start 2023-12-16T21:57:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c23774640 1 -- start start 2023-12-16T21:57:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c23774640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c1c0715b0 0x7f6c1c0cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c23774640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c0ce840 0x7f6c1c163a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c23774640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c1c072b80 con 0x7f6c1c15a450 2023-12-16T21:57:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c23774640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c1c0cecd0 con 0x7f6c1c0715b0 2023-12-16T21:57:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c23774640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c0b2fd0 con 0x7f6c1c0ce840 2023-12-16T21:57:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c21510640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c1c0715b0 0x7f6c1c0cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c21510640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c1c0715b0 0x7f6c1c0cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46154/0 (socket says 172.21.15.28:46154) 2023-12-16T21:57:50.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c21510640 1 -- 172.21.15.28:0/4087649323 learned_addr learned my addr 172.21.15.28:0/4087649323 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:50.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c21d11640 1 --2- 172.21.15.28:0/4087649323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c0ce840 0x7f6c1c163a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c21510640 1 -- 172.21.15.28:0/4087649323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c0ce840 msgr2=0x7f6c1c163a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:50.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c21510640 1 --2- 172.21.15.28:0/4087649323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c0ce840 0x7f6c1c163a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c21510640 1 -- 172.21.15.28:0/4087649323 >> v1:172.21.15.122:6789/0 conn(0x7f6c1c15a450 legacy=0x7f6c1c071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.004+0000 7f6c21510640 1 -- 172.21.15.28:0/4087649323 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1c0b32b0 con 0x7f6c1c0715b0 2023-12-16T21:57:50.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 --2- 172.21.15.28:0/4087649323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c0cf860 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6c18009870 tx=0x7f6c1801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397bbb5deea6016f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c13fff640 1 -- 172.21.15.28:0/4087649323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c180aa670 con 0x7f6c1c0715b0 2023-12-16T21:57:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c13fff640 1 -- 172.21.15.28:0/4087649323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c180aa7d0 con 0x7f6c1c0715b0 2023-12-16T21:57:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c13fff640 1 -- 172.21.15.28:0/4087649323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c180aab60 con 0x7f6c1c0715b0 2023-12-16T21:57:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- 172.21.15.28:0/4087649323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 msgr2=0x7f6c1c0cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 --2- 172.21.15.28:0/4087649323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c0cf860 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6c18009870 tx=0x7f6c1801c510 comp rx=0 tx=0).stop 2023-12-16T21:57:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- 172.21.15.28:0/4087649323 shutdown_connections 2023-12-16T21:57:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 --2- 172.21.15.28:0/4087649323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c0ce840 0x7f6c1c163a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 --2- 172.21.15.28:0/4087649323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c0cf860 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- 172.21.15.28:0/4087649323 >> 172.21.15.28:0/4087649323 conn(0x7f6c1c06d530 msgr2=0x7f6c1c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- 172.21.15.28:0/4087649323 shutdown_connections 2023-12-16T21:57:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- 172.21.15.28:0/4087649323 wait complete. 2023-12-16T21:57:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 Processor -- start 2023-12-16T21:57:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- start start 2023-12-16T21:57:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c1c0ce840 0x7f6c1c1652c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c15a450 0x7f6c1c13f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c1565d0 con 0x7f6c1c0715b0 2023-12-16T21:57:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c138260 con 0x7f6c1c0ce840 2023-12-16T21:57:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c23774640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1c138540 con 0x7f6c1c15a450 2023-12-16T21:57:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c156090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46164/0 (socket says 172.21.15.28:46164) 2023-12-16T21:57:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 -- 172.21.15.28:0/2988534058 learned_addr learned my addr 172.21.15.28:0/2988534058 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 -- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c15a450 msgr2=0x7f6c1c13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c15a450 0x7f6c1c13f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 -- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c1c0ce840 msgr2=0x7f6c1c1652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c1c0ce840 0x7f6c1c1652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.008+0000 7f6c21510640 1 -- 172.21.15.28:0/2988534058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c180094d0 con 0x7f6c1c0715b0 2023-12-16T21:57:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c21510640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c156090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6c1801cdb0 tx=0x7f6c1801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c11ffb640 1 -- 172.21.15.28:0/2988534058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c180aa670 con 0x7f6c1c0715b0 2023-12-16T21:57:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c11ffb640 1 -- 172.21.15.28:0/2988534058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c180b3800 con 0x7f6c1c0715b0 2023-12-16T21:57:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c138820 con 0x7f6c1c0715b0 2023-12-16T21:57:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c11ffb640 1 -- 172.21.15.28:0/2988534058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c180b3b90 con 0x7f6c1c0715b0 2023-12-16T21:57:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c1c138de0 con 0x7f6c1c0715b0 2023-12-16T21:57:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c1c138fd0 con 0x7f6c1c0715b0 2023-12-16T21:57:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c11ffb640 1 -- 172.21.15.28:0/2988534058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6c180b3cf0 con 0x7f6c1c0715b0 2023-12-16T21:57:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c11ffb640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c00066470 0x7f6c00068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c11ffb640 1 -- 172.21.15.28:0/2988534058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c1811ec70 con 0x7f6c1c0715b0 2023-12-16T21:57:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.012+0000 7f6c20d0f640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c00066470 0x7f6c00068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.016+0000 7f6c20d0f640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c00066470 0x7f6c00068930 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c000f70 tx=0x7f6c0c0079c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.016+0000 7f6c11ffb640 1 -- 172.21.15.28:0/2988534058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6c180ec180 con 0x7f6c1c0715b0 2023-12-16T21:57:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.164+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6c1c06a530 con 0x7f6c00066470 2023-12-16T21:57:50.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.180+0000 7f6c11ffb640 1 -- 172.21.15.28:0/2988534058 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264601 (secure 0 0 0) 0x7f6c0c005080 con 0x7f6c00066470 2023-12-16T21:57:50.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:57:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c00066470 msgr2=0x7f6c00068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c00066470 0x7f6c00068930 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c000f70 tx=0x7f6c0c0079c0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 msgr2=0x7f6c1c156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c156090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6c1801cdb0 tx=0x7f6c1801ad50 comp rx=0 tx=0).stop 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 shutdown_connections 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c00066470 0x7f6c00068930 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c1c15a450 0x7f6c1c13f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c1c0ce840 0x7f6c1c1652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 --2- 172.21.15.28:0/2988534058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1c0715b0 0x7f6c1c156090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 >> 172.21.15.28:0/2988534058 conn(0x7f6c1c06d530 msgr2=0x7f6c1c0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 shutdown_connections 2023-12-16T21:57:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.184+0000 7f6c23774640 1 -- 172.21.15.28:0/2988534058 wait complete. 2023-12-16T21:57:50.202 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-12-16T21:57:49.383589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235824,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738896,"statfs":{"total":773094113280,"available":772852629504,"internally_reserved":0,"allocated":3088384,"data_stored":1823980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.906281"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264489+0000","last_change":"2023-12-16T21:57:43.518611+0000","last_active":"2023-12-16T21:57:45.264489+0000","last_peered":"2023-12-16T21:57:45.264489+0000","last_clean":"2023-12-16T21:57:45.264489+0000","last_became_active":"2023-12-16T21:57:43.517704+0000","last_became_peered":"2023-12-16T21:57:43.517704+0000","last_unstale":"2023-12-16T21:57:45.264489+0000","last_undegraded":"2023-12-16T21:57:45.264489+0000","last_fullsized":"2023-12-16T21:57:45.264489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:49:13.751852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870153+0000","last_change":"2023-12-16T21:57:43.485837+0000","last_active":"2023-12-16T21:57:44.870153+0000","last_peered":"2023-12-16T21:57:44.870153+0000","last_clean":"2023-12-16T21:57:44.870153+0000","last_became_active":"2023-12-16T21:57:43.484881+0000","last_became_peered":"2023-12-16T21:57:43.484881+0000","last_unstale":"2023-12-16T21:57:44.870153+0000","last_undegraded":"2023-12-16T21:57:44.870153+0000","last_fullsized":"2023-12-16T21:57:44.870153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252805+0000","last_change":"2023-12-16T21:57:43.492016+0000","last_active":"2023-12-16T21:57:45.252805+0000","last_peered":"2023-12-16T21:57:45.252805+0000","last_clean":"2023-12-16T21:57:45.252805+0000","last_became_active":"2023-12-16T21:57:43.491114+0000","last_became_peered":"2023-12-16T21:57:43.491114+0000","last_unstale":"2023-12-16T21:57:45.252805+0000","last_undegraded":"2023-12-16T21:57:45.252805+0000","last_fullsized":"2023-12-16T21:57:45.252805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262086+0000","last_change":"2023-12-16T21:57:43.510356+0000","last_active":"2023-12-16T21:57:45.262086+0000","last_peered":"2023-12-16T21:57:45.262086+0000","last_clean":"2023-12-16T21:57:45.262086+0000","last_became_active":"2023-12-16T21:57:43.509819+0000","last_became_peered":"2023-12-16T21:57:43.509819+0000","last_unstale":"2023-12-16T21:57:45.262086+0000","last_undegraded":"2023-12-16T21:57:45.262086+0000","last_fullsized":"2023-12-16T21:57:45.262086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264164+0000","last_change":"2023-12-16T21:57:43.507126+0000","last_active":"2023-12-16T21:57:45.264164+0000","last_peered":"2023-12-16T21:57:45.264164+0000","last_clean":"2023-12-16T21:57:45.264164+0000","last_became_active":"2023-12-16T21:57:43.506352+0000","last_became_peered":"2023-12-16T21:57:43.506352+0000","last_unstale":"2023-12-16T21:57:45.264164+0000","last_undegraded":"2023-12-16T21:57:45.264164+0000","last_fullsized":"2023-12-16T21:57:45.264164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:04:09.033093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262865+0000","last_change":"2023-12-16T21:57:43.483088+0000","last_active":"2023-12-16T21:57:45.262865+0000","last_peered":"2023-12-16T21:57:45.262865+0000","last_clean":"2023-12-16T21:57:45.262865+0000","last_became_active":"2023-12-16T21:57:43.482278+0000","last_became_peered":"2023-12-16T21:57:43.482278+0000","last_unstale":"2023-12-16T21:57:45.262865+0000","last_undegraded":"2023-12-16T21:57:45.262865+0000","last_fullsized":"2023-12-16T21:57:45.262865+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:15.360923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261922+0000","last_change":"2023-12-16T21:57:43.494598+0000","last_active":"2023-12-16T21:57:45.261922+0000","last_peered":"2023-12-16T21:57:45.261922+0000","last_clean":"2023-12-16T21:57:45.261922+0000","last_became_active":"2023-12-16T21:57:43.493867+0000","last_became_peered":"2023-12-16T21:57:43.493867+0000","last_unstale":"2023-12-16T21:57:45.261922+0000","last_undegraded":"2023-12-16T21:57:45.261922+0000","last_fullsized":"2023-12-16T21:57:45.261922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262085+0000","last_change":"2023-12-16T21:57:43.504864+0000","last_active":"2023-12-16T21:57:45.262085+0000","last_peered":"2023-12-16T21:57:45.262085+0000","last_clean":"2023-12-16T21:57:45.262085+0000","last_became_active":"2023-12-16T21:57:43.504140+0000","last_became_peered":"2023-12-16T21:57:43.504140+0000","last_unstale":"2023-12-16T21:57:45.262085+0000","last_undegraded":"2023-12-16T21:57:45.262085+0000","last_fullsized":"2023-12-16T21:57:45.262085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:22.190906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264734+0000","last_change":"2023-12-16T21:57:43.518225+0000","last_active":"2023-12-16T21:57:45.264734+0000","last_peered":"2023-12-16T21:57:45.264734+0000","last_clean":"2023-12-16T21:57:45.264734+0000","last_became_active":"2023-12-16T21:57:43.517602+0000","last_became_peered":"2023-12-16T21:57:43.517602+0000","last_unstale":"2023-12-16T21:57:45.264734+0000","last_undegraded":"2023-12-16T21:57:45.264734+0000","last_fullsized":"2023-12-16T21:57:45.264734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:46:04.514997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.460177+0000","last_change":"2023-12-16T21:57:43.468564+0000","last_active":"2023-12-16T21:57:44.460177+0000","last_peered":"2023-12-16T21:57:44.460177+0000","last_clean":"2023-12-16T21:57:44.460177+0000","last_became_active":"2023-12-16T21:57:43.467674+0000","last_became_peered":"2023-12-16T21:57:43.467674+0000","last_unstale":"2023-12-16T21:57:44.460177+0000","last_undegraded":"2023-12-16T21:57:44.460177+0000","last_fullsized":"2023-12-16T21:57:44.460177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458605+0000","last_change":"2023-12-16T21:57:43.474153+0000","last_active":"2023-12-16T21:57:44.458605+0000","last_peered":"2023-12-16T21:57:44.458605+0000","last_clean":"2023-12-16T21:57:44.458605+0000","last_became_active":"2023-12-16T21:57:43.467788+0000","last_became_peered":"2023-12-16T21:57:43.467788+0000","last_unstale":"2023-12-16T21:57:44.458605+0000","last_undegraded":"2023-12-16T21:57:44.458605+0000","last_fullsized":"2023-12-16T21:57:44.458605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458900+0000","last_change":"2023-12-16T21:57:43.494364+0000","last_active":"2023-12-16T21:57:44.458900+0000","last_peered":"2023-12-16T21:57:44.458900+0000","last_clean":"2023-12-16T21:57:44.458900+0000","last_became_active":"2023-12-16T21:57:43.491952+0000","last_became_peered":"2023-12-16T21:57:43.491952+0000","last_unstale":"2023-12-16T21:57:44.458900+0000","last_undegraded":"2023-12-16T21:57:44.458900+0000","last_fullsized":"2023-12-16T21:57:44.458900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:16.932172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262657+0000","last_change":"2023-12-16T21:57:43.490348+0000","last_active":"2023-12-16T21:57:45.262657+0000","last_peered":"2023-12-16T21:57:45.262657+0000","last_clean":"2023-12-16T21:57:45.262657+0000","last_became_active":"2023-12-16T21:57:43.489582+0000","last_became_peered":"2023-12-16T21:57:43.489582+0000","last_unstale":"2023-12-16T21:57:45.262657+0000","last_undegraded":"2023-12-16T21:57:45.262657+0000","last_fullsized":"2023-12-16T21:57:45.262657+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252238+0000","last_change":"2023-12-16T21:57:43.481446+0000","last_active":"2023-12-16T21:57:45.252238+0000","last_peered":"2023-12-16T21:57:45.252238+0000","last_clean":"2023-12-16T21:57:45.252238+0000","last_became_active":"2023-12-16T21:57:43.480422+0000","last_became_peered":"2023-12-16T21:57:43.480422+0000","last_unstale":"2023-12-16T21:57:45.252238+0000","last_undegraded":"2023-12-16T21:57:45.252238+0000","last_fullsized":"2023-12-16T21:57:45.252238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869559+0000","last_change":"2023-12-16T21:57:43.507792+0000","last_active":"2023-12-16T21:57:44.869559+0000","last_peered":"2023-12-16T21:57:44.869559+0000","last_clean":"2023-12-16T21:57:44.869559+0000","last_became_active":"2023-12-16T21:57:43.507030+0000","last_became_peered":"2023-12-16T21:57:43.507030+0000","last_unstale":"2023-12-16T21:57:44.869559+0000","last_undegraded":"2023-12-16T21:57:44.869559+0000","last_fullsized":"2023-12-16T21:57:44.869559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261659+0000","last_change":"2023-12-16T21:57:43.485531+0000","last_active":"2023-12-16T21:57:45.261659+0000","last_peered":"2023-12-16T21:57:45.261659+0000","last_clean":"2023-12-16T21:57:45.261659+0000","last_became_active":"2023-12-16T21:57:43.482538+0000","last_became_peered":"2023-12-16T21:57:43.482538+0000","last_unstale":"2023-12-16T21:57:45.261659+0000","last_undegraded":"2023-12-16T21:57:45.261659+0000","last_fullsized":"2023-12-16T21:57:45.261659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263056+0000","last_change":"2023-12-16T21:57:43.507572+0000","last_active":"2023-12-16T21:57:45.263056+0000","last_peered":"2023-12-16T21:57:45.263056+0000","last_clean":"2023-12-16T21:57:45.263056+0000","last_became_active":"2023-12-16T21:57:43.507072+0000","last_became_peered":"2023-12-16T21:57:43.507072+0000","last_unstale":"2023-12-16T21:57:45.263056+0000","last_undegraded":"2023-12-16T21:57:45.263056+0000","last_fullsized":"2023-12-16T21:57:45.263056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:49:13.751852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.456131+0000","last_change":"2023-12-16T21:57:43.487657+0000","last_active":"2023-12-16T21:57:44.456131+0000","last_peered":"2023-12-16T21:57:44.456131+0000","last_clean":"2023-12-16T21:57:44.456131+0000","last_became_active":"2023-12-16T21:57:43.484848+0000","last_became_peered":"2023-12-16T21:57:43.484848+0000","last_unstale":"2023-12-16T21:57:44.456131+0000","last_undegraded":"2023-12-16T21:57:44.456131+0000","last_fullsized":"2023-12-16T21:57:44.456131+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:14:59.932149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264990+0000","last_change":"2023-12-16T21:57:43.503136+0000","last_active":"2023-12-16T21:57:45.264990+0000","last_peered":"2023-12-16T21:57:45.264990+0000","last_clean":"2023-12-16T21:57:45.264990+0000","last_became_active":"2023-12-16T21:57:43.502321+0000","last_became_peered":"2023-12-16T21:57:43.502321+0000","last_unstale":"2023-12-16T21:57:45.264990+0000","last_undegraded":"2023-12-16T21:57:45.264990+0000","last_fullsized":"2023-12-16T21:57:45.264990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:29.457727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458117+0000","last_change":"2023-12-16T21:57:43.496151+0000","last_active":"2023-12-16T21:57:44.458117+0000","last_peered":"2023-12-16T21:57:44.458117+0000","last_clean":"2023-12-16T21:57:44.458117+0000","last_became_active":"2023-12-16T21:57:43.495293+0000","last_became_peered":"2023-12-16T21:57:43.495293+0000","last_unstale":"2023-12-16T21:57:44.458117+0000","last_undegraded":"2023-12-16T21:57:44.458117+0000","last_fullsized":"2023-12-16T21:57:44.458117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252271+0000","last_change":"2023-12-16T21:57:43.507839+0000","last_active":"2023-12-16T21:57:45.252271+0000","last_peered":"2023-12-16T21:57:45.252271+0000","last_clean":"2023-12-16T21:57:45.252271+0000","last_became_active":"2023-12-16T21:57:43.507150+0000","last_became_peered":"2023-12-16T21:57:43.507150+0000","last_unstale":"2023-12-16T21:57:45.252271+0000","last_undegraded":"2023-12-16T21:57:45.252271+0000","last_fullsized":"2023-12-16T21:57:45.252271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263163+0000","last_change":"2023-12-16T21:57:43.487735+0000","last_active":"2023-12-16T21:57:45.263163+0000","last_peered":"2023-12-16T21:57:45.263163+0000","last_clean":"2023-12-16T21:57:45.263163+0000","last_became_active":"2023-12-16T21:57:43.486972+0000","last_became_peered":"2023-12-16T21:57:43.486972+0000","last_unstale":"2023-12-16T21:57:45.263163+0000","last_undegraded":"2023-12-16T21:57:45.263163+0000","last_fullsized":"2023-12-16T21:57:45.263163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:46:04.514997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262348+0000","last_change":"2023-12-16T21:57:43.513648+0000","last_active":"2023-12-16T21:57:45.262348+0000","last_peered":"2023-12-16T21:57:45.262348+0000","last_clean":"2023-12-16T21:57:45.262348+0000","last_became_active":"2023-12-16T21:57:43.513086+0000","last_became_peered":"2023-12-16T21:57:43.513086+0000","last_unstale":"2023-12-16T21:57:45.262348+0000","last_undegraded":"2023-12-16T21:57:45.262348+0000","last_fullsized":"2023-12-16T21:57:45.262348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261798+0000","last_change":"2023-12-16T21:57:43.505290+0000","last_active":"2023-12-16T21:57:45.261798+0000","last_peered":"2023-12-16T21:57:45.261798+0000","last_clean":"2023-12-16T21:57:45.261798+0000","last_became_active":"2023-12-16T21:57:43.504600+0000","last_became_peered":"2023-12-16T21:57:43.504600+0000","last_unstale":"2023-12-16T21:57:45.261798+0000","last_undegraded":"2023-12-16T21:57:45.261798+0000","last_fullsized":"2023-12-16T21:57:45.261798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263041+0000","last_change":"2023-12-16T21:57:43.510116+0000","last_active":"2023-12-16T21:57:45.263041+0000","last_peered":"2023-12-16T21:57:45.263041+0000","last_clean":"2023-12-16T21:57:45.263041+0000","last_became_active":"2023-12-16T21:57:43.509260+0000","last_became_peered":"2023-12-16T21:57:43.509260+0000","last_unstale":"2023-12-16T21:57:45.263041+0000","last_undegraded":"2023-12-16T21:57:45.263041+0000","last_fullsized":"2023-12-16T21:57:45.263041+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.460344+0000","last_change":"2023-12-16T21:57:43.489237+0000","last_active":"2023-12-16T21:57:44.460344+0000","last_peered":"2023-12-16T21:57:44.460344+0000","last_clean":"2023-12-16T21:57:44.460344+0000","last_became_active":"2023-12-16T21:57:43.488252+0000","last_became_peered":"2023-12-16T21:57:43.488252+0000","last_unstale":"2023-12-16T21:57:44.460344+0000","last_undegraded":"2023-12-16T21:57:44.460344+0000","last_fullsized":"2023-12-16T21:57:44.460344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869771+0000","last_change":"2023-12-16T21:57:43.502339+0000","last_active":"2023-12-16T21:57:44.869771+0000","last_peered":"2023-12-16T21:57:44.869771+0000","last_clean":"2023-12-16T21:57:44.869771+0000","last_became_active":"2023-12-16T21:57:43.501434+0000","last_became_peered":"2023-12-16T21:57:43.501434+0000","last_unstale":"2023-12-16T21:57:44.869771+0000","last_undegraded":"2023-12-16T21:57:44.869771+0000","last_fullsized":"2023-12-16T21:57:44.869771+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263170+0000","last_change":"2023-12-16T21:57:43.509617+0000","last_active":"2023-12-16T21:57:45.263170+0000","last_peered":"2023-12-16T21:57:45.263170+0000","last_clean":"2023-12-16T21:57:45.263170+0000","last_became_active":"2023-12-16T21:57:43.508922+0000","last_became_peered":"2023-12-16T21:57:43.508922+0000","last_unstale":"2023-12-16T21:57:45.263170+0000","last_undegraded":"2023-12-16T21:57:45.263170+0000","last_fullsized":"2023-12-16T21:57:45.263170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252408+0000","last_change":"2023-12-16T21:57:43.505933+0000","last_active":"2023-12-16T21:57:45.252408+0000","last_peered":"2023-12-16T21:57:45.252408+0000","last_clean":"2023-12-16T21:57:45.252408+0000","last_became_active":"2023-12-16T21:57:43.504463+0000","last_became_peered":"2023-12-16T21:57:43.504463+0000","last_unstale":"2023-12-16T21:57:45.252408+0000","last_undegraded":"2023-12-16T21:57:45.252408+0000","last_fullsized":"2023-12-16T21:57:45.252408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263332+0000","last_change":"2023-12-16T21:57:43.501738+0000","last_active":"2023-12-16T21:57:45.263332+0000","last_peered":"2023-12-16T21:57:45.263332+0000","last_clean":"2023-12-16T21:57:45.263332+0000","last_became_active":"2023-12-16T21:57:43.501014+0000","last_became_peered":"2023-12-16T21:57:43.501014+0000","last_unstale":"2023-12-16T21:57:45.263332+0000","last_undegraded":"2023-12-16T21:57:45.263332+0000","last_fullsized":"2023-12-16T21:57:45.263332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:47:38.932092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.456015+0000","last_change":"2023-12-16T21:57:43.477348+0000","last_active":"2023-12-16T21:57:44.456015+0000","last_peered":"2023-12-16T21:57:44.456015+0000","last_clean":"2023-12-16T21:57:44.456015+0000","last_became_active":"2023-12-16T21:57:43.476616+0000","last_became_peered":"2023-12-16T21:57:43.476616+0000","last_unstale":"2023-12-16T21:57:44.456015+0000","last_undegraded":"2023-12-16T21:57:44.456015+0000","last_fullsized":"2023-12-16T21:57:44.456015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:18:16.866270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457542+0000","last_change":"2023-12-16T21:57:43.483725+0000","last_active":"2023-12-16T21:57:44.457542+0000","last_peered":"2023-12-16T21:57:44.457542+0000","last_clean":"2023-12-16T21:57:44.457542+0000","last_became_active":"2023-12-16T21:57:43.482870+0000","last_became_peered":"2023-12-16T21:57:43.482870+0000","last_unstale":"2023-12-16T21:57:44.457542+0000","last_undegraded":"2023-12-16T21:57:44.457542+0000","last_fullsized":"2023-12-16T21:57:44.457542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458048+0000","last_change":"2023-12-16T21:57:43.492281+0000","last_active":"2023-12-16T21:57:44.458048+0000","last_peered":"2023-12-16T21:57:44.458048+0000","last_clean":"2023-12-16T21:57:44.458048+0000","last_became_active":"2023-12-16T21:57:43.491538+0000","last_became_peered":"2023-12-16T21:57:43.491538+0000","last_unstale":"2023-12-16T21:57:44.458048+0000","last_undegraded":"2023-12-16T21:57:44.458048+0000","last_fullsized":"2023-12-16T21:57:44.458048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:21:57.846367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262966+0000","last_change":"2023-12-16T21:57:42.522074+0000","last_active":"2023-12-16T21:57:45.262966+0000","last_peered":"2023-12-16T21:57:45.262966+0000","last_clean":"2023-12-16T21:57:45.262966+0000","last_became_active":"2023-12-16T21:57:42.521769+0000","last_became_peered":"2023-12-16T21:57:42.521769+0000","last_unstale":"2023-12-16T21:57:45.262966+0000","last_undegraded":"2023-12-16T21:57:45.262966+0000","last_fullsized":"2023-12-16T21:57:45.262966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:42:59.000744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252714+0000","last_change":"2023-12-16T21:57:43.495840+0000","last_active":"2023-12-16T21:57:45.252714+0000","last_peered":"2023-12-16T21:57:45.252714+0000","last_clean":"2023-12-16T21:57:45.252714+0000","last_became_active":"2023-12-16T21:57:43.495114+0000","last_became_peered":"2023-12-16T21:57:43.495114+0000","last_unstale":"2023-12-16T21:57:45.252714+0000","last_undegraded":"2023-12-16T21:57:45.252714+0000","last_fullsized":"2023-12-16T21:57:45.252714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458001+0000","last_change":"2023-12-16T21:57:42.515225+0000","last_active":"2023-12-16T21:57:44.458001+0000","last_peered":"2023-12-16T21:57:44.458001+0000","last_clean":"2023-12-16T21:57:44.458001+0000","last_became_active":"2023-12-16T21:57:42.514509+0000","last_became_peered":"2023-12-16T21:57:42.514509+0000","last_unstale":"2023-12-16T21:57:44.458001+0000","last_undegraded":"2023-12-16T21:57:44.458001+0000","last_fullsized":"2023-12-16T21:57:44.458001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:21.179733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262272+0000","last_change":"2023-12-16T21:57:42.515658+0000","last_active":"2023-12-16T21:57:45.262272+0000","last_peered":"2023-12-16T21:57:45.262272+0000","last_clean":"2023-12-16T21:57:45.262272+0000","last_became_active":"2023-12-16T21:57:42.515275+0000","last_became_peered":"2023-12-16T21:57:42.515275+0000","last_unstale":"2023-12-16T21:57:45.262272+0000","last_undegraded":"2023-12-16T21:57:45.262272+0000","last_fullsized":"2023-12-16T21:57:45.262272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:00:29.774694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262293+0000","last_change":"2023-12-16T21:57:43.493987+0000","last_active":"2023-12-16T21:57:45.262293+0000","last_peered":"2023-12-16T21:57:45.262293+0000","last_clean":"2023-12-16T21:57:45.262293+0000","last_became_active":"2023-12-16T21:57:43.493193+0000","last_became_peered":"2023-12-16T21:57:43.493193+0000","last_unstale":"2023-12-16T21:57:45.262293+0000","last_undegraded":"2023-12-16T21:57:45.262293+0000","last_fullsized":"2023-12-16T21:57:45.262293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261893+0000","last_change":"2023-12-16T21:57:42.508909+0000","last_active":"2023-12-16T21:57:45.261893+0000","last_peered":"2023-12-16T21:57:45.261893+0000","last_clean":"2023-12-16T21:57:45.261893+0000","last_became_active":"2023-12-16T21:57:42.508250+0000","last_became_peered":"2023-12-16T21:57:42.508250+0000","last_unstale":"2023-12-16T21:57:45.261893+0000","last_undegraded":"2023-12-16T21:57:45.261893+0000","last_fullsized":"2023-12-16T21:57:45.261893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:39.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869537+0000","last_change":"2023-12-16T21:57:43.487695+0000","last_active":"2023-12-16T21:57:44.869537+0000","last_peered":"2023-12-16T21:57:44.869537+0000","last_clean":"2023-12-16T21:57:44.869537+0000","last_became_active":"2023-12-16T21:57:43.486936+0000","last_became_peered":"2023-12-16T21:57:43.486936+0000","last_unstale":"2023-12-16T21:57:44.869537+0000","last_undegraded":"2023-12-16T21:57:44.869537+0000","last_fullsized":"2023-12-16T21:57:44.869537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458471+0000","last_change":"2023-12-16T21:57:42.526210+0000","last_active":"2023-12-16T21:57:44.458471+0000","last_peered":"2023-12-16T21:57:44.458471+0000","last_clean":"2023-12-16T21:57:44.458471+0000","last_became_active":"2023-12-16T21:57:42.525460+0000","last_became_peered":"2023-12-16T21:57:42.525460+0000","last_unstale":"2023-12-16T21:57:44.458471+0000","last_undegraded":"2023-12-16T21:57:44.458471+0000","last_fullsized":"2023-12-16T21:57:44.458471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:43:36.373508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263137+0000","last_change":"2023-12-16T21:57:43.483521+0000","last_active":"2023-12-16T21:57:45.263137+0000","last_peered":"2023-12-16T21:57:45.263137+0000","last_clean":"2023-12-16T21:57:45.263137+0000","last_became_active":"2023-12-16T21:57:43.482802+0000","last_became_peered":"2023-12-16T21:57:43.482802+0000","last_unstale":"2023-12-16T21:57:45.263137+0000","last_undegraded":"2023-12-16T21:57:45.263137+0000","last_fullsized":"2023-12-16T21:57:45.263137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:29.175289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263361+0000","last_change":"2023-12-16T21:57:42.517029+0000","last_active":"2023-12-16T21:57:45.263361+0000","last_peered":"2023-12-16T21:57:45.263361+0000","last_clean":"2023-12-16T21:57:45.263361+0000","last_became_active":"2023-12-16T21:57:42.516592+0000","last_became_peered":"2023-12-16T21:57:42.516592+0000","last_unstale":"2023-12-16T21:57:45.263361+0000","last_undegraded":"2023-12-16T21:57:45.263361+0000","last_fullsized":"2023-12-16T21:57:45.263361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:42:59.000744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252990+0000","last_change":"2023-12-16T21:57:43.480932+0000","last_active":"2023-12-16T21:57:45.252990+0000","last_peered":"2023-12-16T21:57:45.252990+0000","last_clean":"2023-12-16T21:57:45.252990+0000","last_became_active":"2023-12-16T21:57:43.480056+0000","last_became_peered":"2023-12-16T21:57:43.480056+0000","last_unstale":"2023-12-16T21:57:45.252990+0000","last_undegraded":"2023-12-16T21:57:45.252990+0000","last_fullsized":"2023-12-16T21:57:45.252990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869739+0000","last_change":"2023-12-16T21:57:42.499117+0000","last_active":"2023-12-16T21:57:44.869739+0000","last_peered":"2023-12-16T21:57:44.869739+0000","last_clean":"2023-12-16T21:57:44.869739+0000","last_became_active":"2023-12-16T21:57:42.498315+0000","last_became_peered":"2023-12-16T21:57:42.498315+0000","last_unstale":"2023-12-16T21:57:44.869739+0000","last_undegraded":"2023-12-16T21:57:44.869739+0000","last_fullsized":"2023-12-16T21:57:44.869739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:14:36.842626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459102+0000","last_change":"2023-12-16T21:57:43.507811+0000","last_active":"2023-12-16T21:57:44.459102+0000","last_peered":"2023-12-16T21:57:44.459102+0000","last_clean":"2023-12-16T21:57:44.459102+0000","last_became_active":"2023-12-16T21:57:43.507089+0000","last_became_peered":"2023-12-16T21:57:43.507089+0000","last_unstale":"2023-12-16T21:57:44.459102+0000","last_undegraded":"2023-12-16T21:57:44.459102+0000","last_fullsized":"2023-12-16T21:57:44.459102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:43:00.011918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458372+0000","last_change":"2023-12-16T21:57:42.512797+0000","last_active":"2023-12-16T21:57:44.458372+0000","last_peered":"2023-12-16T21:57:44.458372+0000","last_clean":"2023-12-16T21:57:44.458372+0000","last_became_active":"2023-12-16T21:57:42.512055+0000","last_became_peered":"2023-12-16T21:57:42.512055+0000","last_unstale":"2023-12-16T21:57:44.458372+0000","last_undegraded":"2023-12-16T21:57:44.458372+0000","last_fullsized":"2023-12-16T21:57:44.458372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:15.920999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264337+0000","last_change":"2023-12-16T21:57:43.510475+0000","last_active":"2023-12-16T21:57:45.264337+0000","last_peered":"2023-12-16T21:57:45.264337+0000","last_clean":"2023-12-16T21:57:45.264337+0000","last_became_active":"2023-12-16T21:57:43.509720+0000","last_became_peered":"2023-12-16T21:57:43.509720+0000","last_unstale":"2023-12-16T21:57:45.264337+0000","last_undegraded":"2023-12-16T21:57:45.264337+0000","last_fullsized":"2023-12-16T21:57:45.264337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:15.360923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252634+0000","last_change":"2023-12-16T21:57:42.518031+0000","last_active":"2023-12-16T21:57:45.252634+0000","last_peered":"2023-12-16T21:57:45.252634+0000","last_clean":"2023-12-16T21:57:45.252634+0000","last_became_active":"2023-12-16T21:57:42.517664+0000","last_became_peered":"2023-12-16T21:57:42.517664+0000","last_unstale":"2023-12-16T21:57:45.252634+0000","last_undegraded":"2023-12-16T21:57:45.252634+0000","last_fullsized":"2023-12-16T21:57:45.252634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252215+0000","last_change":"2023-12-16T21:57:43.493896+0000","last_active":"2023-12-16T21:57:45.252215+0000","last_peered":"2023-12-16T21:57:45.252215+0000","last_clean":"2023-12-16T21:57:45.252215+0000","last_became_active":"2023-12-16T21:57:43.492917+0000","last_became_peered":"2023-12-16T21:57:43.492917+0000","last_unstale":"2023-12-16T21:57:45.252215+0000","last_undegraded":"2023-12-16T21:57:45.252215+0000","last_fullsized":"2023-12-16T21:57:45.252215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:22.190906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263439+0000","last_change":"2023-12-16T21:57:42.517161+0000","last_active":"2023-12-16T21:57:45.263439+0000","last_peered":"2023-12-16T21:57:45.263439+0000","last_clean":"2023-12-16T21:57:45.263439+0000","last_became_active":"2023-12-16T21:57:42.516388+0000","last_became_peered":"2023-12-16T21:57:42.516388+0000","last_unstale":"2023-12-16T21:57:45.263439+0000","last_undegraded":"2023-12-16T21:57:45.263439+0000","last_fullsized":"2023-12-16T21:57:45.263439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:15.423681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264304+0000","last_change":"2023-12-16T21:57:43.512923+0000","last_active":"2023-12-16T21:57:45.264304+0000","last_peered":"2023-12-16T21:57:45.264304+0000","last_clean":"2023-12-16T21:57:45.264304+0000","last_became_active":"2023-12-16T21:57:43.512254+0000","last_became_peered":"2023-12-16T21:57:43.512254+0000","last_unstale":"2023-12-16T21:57:45.264304+0000","last_undegraded":"2023-12-16T21:57:45.264304+0000","last_fullsized":"2023-12-16T21:57:45.264304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:20.834561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457916+0000","last_change":"2023-12-16T21:57:42.511480+0000","last_active":"2023-12-16T21:57:44.457916+0000","last_peered":"2023-12-16T21:57:44.457916+0000","last_clean":"2023-12-16T21:57:44.457916+0000","last_became_active":"2023-12-16T21:57:42.510772+0000","last_became_peered":"2023-12-16T21:57:42.510772+0000","last_unstale":"2023-12-16T21:57:44.457916+0000","last_undegraded":"2023-12-16T21:57:44.457916+0000","last_fullsized":"2023-12-16T21:57:44.457916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457878+0000","last_change":"2023-12-16T21:57:43.493903+0000","last_active":"2023-12-16T21:57:44.457878+0000","last_peered":"2023-12-16T21:57:44.457878+0000","last_clean":"2023-12-16T21:57:44.457878+0000","last_became_active":"2023-12-16T21:57:43.493169+0000","last_became_peered":"2023-12-16T21:57:43.493169+0000","last_unstale":"2023-12-16T21:57:44.457878+0000","last_undegraded":"2023-12-16T21:57:44.457878+0000","last_fullsized":"2023-12-16T21:57:44.457878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.401674+0000","last_change":"2023-12-16T21:57:42.498884+0000","last_active":"2023-12-16T21:57:44.401674+0000","last_peered":"2023-12-16T21:57:44.401674+0000","last_clean":"2023-12-16T21:57:44.401674+0000","last_became_active":"2023-12-16T21:57:42.498452+0000","last_became_peered":"2023-12-16T21:57:42.498452+0000","last_unstale":"2023-12-16T21:57:44.401674+0000","last_undegraded":"2023-12-16T21:57:44.401674+0000","last_fullsized":"2023-12-16T21:57:44.401674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:37:03.298907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263857+0000","last_change":"2023-12-16T21:57:43.513792+0000","last_active":"2023-12-16T21:57:45.263857+0000","last_peered":"2023-12-16T21:57:45.263857+0000","last_clean":"2023-12-16T21:57:45.263857+0000","last_became_active":"2023-12-16T21:57:43.513001+0000","last_became_peered":"2023-12-16T21:57:43.513001+0000","last_unstale":"2023-12-16T21:57:45.263857+0000","last_undegraded":"2023-12-16T21:57:45.263857+0000","last_fullsized":"2023-12-16T21:57:45.263857+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459156+0000","last_change":"2023-12-16T21:57:42.512676+0000","last_active":"2023-12-16T21:57:44.459156+0000","last_peered":"2023-12-16T21:57:44.459156+0000","last_clean":"2023-12-16T21:57:44.459156+0000","last_became_active":"2023-12-16T21:57:42.512267+0000","last_became_peered":"2023-12-16T21:57:42.512267+0000","last_unstale":"2023-12-16T21:57:44.459156+0000","last_undegraded":"2023-12-16T21:57:44.459156+0000","last_fullsized":"2023-12-16T21:57:44.459156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:23:06.394772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264897+0000","last_change":"2023-12-16T21:57:43.517003+0000","last_active":"2023-12-16T21:57:45.264897+0000","last_peered":"2023-12-16T21:57:45.264897+0000","last_clean":"2023-12-16T21:57:45.264897+0000","last_became_active":"2023-12-16T21:57:43.516197+0000","last_became_peered":"2023-12-16T21:57:43.516197+0000","last_unstale":"2023-12-16T21:57:45.264897+0000","last_undegraded":"2023-12-16T21:57:45.264897+0000","last_fullsized":"2023-12-16T21:57:45.264897+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:48:39.071922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.401840+0000","last_change":"2023-12-16T21:57:42.512911+0000","last_active":"2023-12-16T21:57:44.401840+0000","last_peered":"2023-12-16T21:57:44.401840+0000","last_clean":"2023-12-16T21:57:44.401840+0000","last_became_active":"2023-12-16T21:57:42.512410+0000","last_became_peered":"2023-12-16T21:57:42.512410+0000","last_unstale":"2023-12-16T21:57:44.401840+0000","last_undegraded":"2023-12-16T21:57:44.401840+0000","last_fullsized":"2023-12-16T21:57:44.401840+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:00:58.952738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264838+0000","last_change":"2023-12-16T21:57:43.518693+0000","last_active":"2023-12-16T21:57:45.264838+0000","last_peered":"2023-12-16T21:57:45.264838+0000","last_clean":"2023-12-16T21:57:45.264838+0000","last_became_active":"2023-12-16T21:57:43.518184+0000","last_became_peered":"2023-12-16T21:57:43.518184+0000","last_unstale":"2023-12-16T21:57:45.264838+0000","last_undegraded":"2023-12-16T21:57:45.264838+0000","last_fullsized":"2023-12-16T21:57:45.264838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:47:38.932092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869962+0000","last_change":"2023-12-16T21:57:42.505084+0000","last_active":"2023-12-16T21:57:44.869962+0000","last_peered":"2023-12-16T21:57:44.869962+0000","last_clean":"2023-12-16T21:57:44.869962+0000","last_became_active":"2023-12-16T21:57:42.502948+0000","last_became_peered":"2023-12-16T21:57:42.502948+0000","last_unstale":"2023-12-16T21:57:44.869962+0000","last_undegraded":"2023-12-16T21:57:44.869962+0000","last_fullsized":"2023-12-16T21:57:44.869962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262484+0000","last_change":"2023-12-16T21:57:43.499808+0000","last_active":"2023-12-16T21:57:45.262484+0000","last_peered":"2023-12-16T21:57:45.262484+0000","last_clean":"2023-12-16T21:57:45.262484+0000","last_became_active":"2023-12-16T21:57:43.498922+0000","last_became_peered":"2023-12-16T21:57:43.498922+0000","last_unstale":"2023-12-16T21:57:45.262484+0000","last_undegraded":"2023-12-16T21:57:45.262484+0000","last_fullsized":"2023-12-16T21:57:45.262484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:04:09.033093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.401759+0000","last_change":"2023-12-16T21:57:42.509838+0000","last_active":"2023-12-16T21:57:44.401759+0000","last_peered":"2023-12-16T21:57:44.401759+0000","last_clean":"2023-12-16T21:57:44.401759+0000","last_became_active":"2023-12-16T21:57:42.508994+0000","last_became_peered":"2023-12-16T21:57:42.508994+0000","last_unstale":"2023-12-16T21:57:44.401759+0000","last_undegraded":"2023-12-16T21:57:44.401759+0000","last_fullsized":"2023-12-16T21:57:44.401759+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:03:03.258824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264723+0000","last_change":"2023-12-16T21:57:43.516388+0000","last_active":"2023-12-16T21:57:45.264723+0000","last_peered":"2023-12-16T21:57:45.264723+0000","last_clean":"2023-12-16T21:57:45.264723+0000","last_became_active":"2023-12-16T21:57:43.515749+0000","last_became_peered":"2023-12-16T21:57:43.515749+0000","last_unstale":"2023-12-16T21:57:45.264723+0000","last_undegraded":"2023-12-16T21:57:45.264723+0000","last_fullsized":"2023-12-16T21:57:45.264723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:29.175289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.474023+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:57:43.474023+0000","last_peered":"2023-12-16T21:57:43.474023+0000","last_clean":"2023-12-16T21:57:43.474023+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:57:43.474023+0000","last_undegraded":"2023-12-16T21:57:43.474023+0000","last_fullsized":"2023-12-16T21:57:43.474023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261664+0000","last_change":"2023-12-16T21:57:42.508124+0000","last_active":"2023-12-16T21:57:45.261664+0000","last_peered":"2023-12-16T21:57:45.261664+0000","last_clean":"2023-12-16T21:57:45.261664+0000","last_became_active":"2023-12-16T21:57:42.507727+0000","last_became_peered":"2023-12-16T21:57:42.507727+0000","last_unstale":"2023-12-16T21:57:45.261664+0000","last_undegraded":"2023-12-16T21:57:45.261664+0000","last_fullsized":"2023-12-16T21:57:45.261664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262915+0000","last_change":"2023-12-16T21:57:43.490787+0000","last_active":"2023-12-16T21:57:45.262915+0000","last_peered":"2023-12-16T21:57:45.262915+0000","last_clean":"2023-12-16T21:57:45.262915+0000","last_became_active":"2023-12-16T21:57:43.490013+0000","last_became_peered":"2023-12-16T21:57:43.490013+0000","last_unstale":"2023-12-16T21:57:45.262915+0000","last_undegraded":"2023-12-16T21:57:45.262915+0000","last_fullsized":"2023-12-16T21:57:45.262915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262013+0000","last_change":"2023-12-16T21:57:42.523476+0000","last_active":"2023-12-16T21:57:45.262013+0000","last_peered":"2023-12-16T21:57:45.262013+0000","last_clean":"2023-12-16T21:57:45.262013+0000","last_became_active":"2023-12-16T21:57:42.522441+0000","last_became_peered":"2023-12-16T21:57:42.522441+0000","last_unstale":"2023-12-16T21:57:45.262013+0000","last_undegraded":"2023-12-16T21:57:45.262013+0000","last_fullsized":"2023-12-16T21:57:45.262013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:15.920999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252089+0000","last_change":"2023-12-16T21:57:43.508913+0000","last_active":"2023-12-16T21:57:45.252089+0000","last_peered":"2023-12-16T21:57:45.252089+0000","last_clean":"2023-12-16T21:57:45.252089+0000","last_became_active":"2023-12-16T21:57:43.508133+0000","last_became_peered":"2023-12-16T21:57:43.508133+0000","last_unstale":"2023-12-16T21:57:45.252089+0000","last_undegraded":"2023-12-16T21:57:45.252089+0000","last_fullsized":"2023-12-16T21:57:45.252089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:58.171558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.401737+0000","last_change":"2023-12-16T21:57:42.513117+0000","last_active":"2023-12-16T21:57:44.401737+0000","last_peered":"2023-12-16T21:57:44.401737+0000","last_clean":"2023-12-16T21:57:44.401737+0000","last_became_active":"2023-12-16T21:57:42.512449+0000","last_became_peered":"2023-12-16T21:57:42.512449+0000","last_unstale":"2023-12-16T21:57:44.401737+0000","last_undegraded":"2023-12-16T21:57:44.401737+0000","last_fullsized":"2023-12-16T21:57:44.401737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:55:25.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869573+0000","last_change":"2023-12-16T21:57:43.486971+0000","last_active":"2023-12-16T21:57:44.869573+0000","last_peered":"2023-12-16T21:57:44.869573+0000","last_clean":"2023-12-16T21:57:44.869573+0000","last_became_active":"2023-12-16T21:57:43.483854+0000","last_became_peered":"2023-12-16T21:57:43.483854+0000","last_unstale":"2023-12-16T21:57:44.869573+0000","last_undegraded":"2023-12-16T21:57:44.869573+0000","last_fullsized":"2023-12-16T21:57:44.869573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263608+0000","last_change":"2023-12-16T21:57:42.501969+0000","last_active":"2023-12-16T21:57:45.263608+0000","last_peered":"2023-12-16T21:57:45.263608+0000","last_clean":"2023-12-16T21:57:45.263608+0000","last_became_active":"2023-12-16T21:57:42.501059+0000","last_became_peered":"2023-12-16T21:57:42.501059+0000","last_unstale":"2023-12-16T21:57:45.263608+0000","last_undegraded":"2023-12-16T21:57:45.263608+0000","last_fullsized":"2023-12-16T21:57:45.263608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:09.342194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459101+0000","last_change":"2023-12-16T21:57:43.506507+0000","last_active":"2023-12-16T21:57:44.459101+0000","last_peered":"2023-12-16T21:57:44.459101+0000","last_clean":"2023-12-16T21:57:44.459101+0000","last_became_active":"2023-12-16T21:57:43.505685+0000","last_became_peered":"2023-12-16T21:57:43.505685+0000","last_unstale":"2023-12-16T21:57:44.459101+0000","last_undegraded":"2023-12-16T21:57:44.459101+0000","last_fullsized":"2023-12-16T21:57:44.459101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:40.013144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869952+0000","last_change":"2023-12-16T21:57:42.503616+0000","last_active":"2023-12-16T21:57:44.869952+0000","last_peered":"2023-12-16T21:57:44.869952+0000","last_clean":"2023-12-16T21:57:44.869952+0000","last_became_active":"2023-12-16T21:57:42.502114+0000","last_became_peered":"2023-12-16T21:57:42.502114+0000","last_unstale":"2023-12-16T21:57:44.869952+0000","last_undegraded":"2023-12-16T21:57:44.869952+0000","last_fullsized":"2023-12-16T21:57:44.869952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:57.160385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262539+0000","last_change":"2023-12-16T21:57:43.500130+0000","last_active":"2023-12-16T21:57:45.262539+0000","last_peered":"2023-12-16T21:57:45.262539+0000","last_clean":"2023-12-16T21:57:45.262539+0000","last_became_active":"2023-12-16T21:57:43.499128+0000","last_became_peered":"2023-12-16T21:57:43.499128+0000","last_unstale":"2023-12-16T21:57:45.262539+0000","last_undegraded":"2023-12-16T21:57:45.262539+0000","last_fullsized":"2023-12-16T21:57:45.262539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:20.834561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.265035+0000","last_change":"2023-12-16T21:57:42.526337+0000","last_active":"2023-12-16T21:57:45.265035+0000","last_peered":"2023-12-16T21:57:45.265035+0000","last_clean":"2023-12-16T21:57:45.265035+0000","last_became_active":"2023-12-16T21:57:42.525796+0000","last_became_peered":"2023-12-16T21:57:42.525796+0000","last_unstale":"2023-12-16T21:57:45.265035+0000","last_undegraded":"2023-12-16T21:57:45.265035+0000","last_fullsized":"2023-12-16T21:57:45.265035+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:00:29.774694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870185+0000","last_change":"2023-12-16T21:57:43.477509+0000","last_active":"2023-12-16T21:57:44.870185+0000","last_peered":"2023-12-16T21:57:44.870185+0000","last_clean":"2023-12-16T21:57:44.870185+0000","last_became_active":"2023-12-16T21:57:43.476854+0000","last_became_peered":"2023-12-16T21:57:43.476854+0000","last_unstale":"2023-12-16T21:57:44.870185+0000","last_undegraded":"2023-12-16T21:57:44.870185+0000","last_fullsized":"2023-12-16T21:57:44.870185+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.401085+0000","last_change":"2023-12-16T21:57:42.499802+0000","last_active":"2023-12-16T21:57:45.401085+0000","last_peered":"2023-12-16T21:57:45.401085+0000","last_clean":"2023-12-16T21:57:45.401085+0000","last_became_active":"2023-12-16T21:57:42.498817+0000","last_became_peered":"2023-12-16T21:57:42.498817+0000","last_unstale":"2023-12-16T21:57:45.401085+0000","last_undegraded":"2023-12-16T21:57:45.401085+0000","last_fullsized":"2023-12-16T21:57:45.401085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263239+0000","last_change":"2023-12-16T21:57:43.490695+0000","last_active":"2023-12-16T21:57:45.263239+0000","last_peered":"2023-12-16T21:57:45.263239+0000","last_clean":"2023-12-16T21:57:45.263239+0000","last_became_active":"2023-12-16T21:57:43.489927+0000","last_became_peered":"2023-12-16T21:57:43.489927+0000","last_unstale":"2023-12-16T21:57:45.263239+0000","last_undegraded":"2023-12-16T21:57:45.263239+0000","last_fullsized":"2023-12-16T21:57:45.263239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458559+0000","last_change":"2023-12-16T21:57:42.523142+0000","last_active":"2023-12-16T21:57:44.458559+0000","last_peered":"2023-12-16T21:57:44.458559+0000","last_clean":"2023-12-16T21:57:44.458559+0000","last_became_active":"2023-12-16T21:57:42.521597+0000","last_became_peered":"2023-12-16T21:57:42.521597+0000","last_unstale":"2023-12-16T21:57:44.458559+0000","last_undegraded":"2023-12-16T21:57:44.458559+0000","last_fullsized":"2023-12-16T21:57:44.458559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252927+0000","last_change":"2023-12-16T21:57:43.483454+0000","last_active":"2023-12-16T21:57:45.252927+0000","last_peered":"2023-12-16T21:57:45.252927+0000","last_clean":"2023-12-16T21:57:45.252927+0000","last_became_active":"2023-12-16T21:57:43.482451+0000","last_became_peered":"2023-12-16T21:57:43.482451+0000","last_unstale":"2023-12-16T21:57:45.252927+0000","last_undegraded":"2023-12-16T21:57:45.252927+0000","last_fullsized":"2023-12-16T21:57:45.252927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261795+0000","last_change":"2023-12-16T21:57:42.522091+0000","last_active":"2023-12-16T21:57:45.261795+0000","last_peered":"2023-12-16T21:57:45.261795+0000","last_clean":"2023-12-16T21:57:45.261795+0000","last_became_active":"2023-12-16T21:57:42.521328+0000","last_became_peered":"2023-12-16T21:57:42.521328+0000","last_unstale":"2023-12-16T21:57:45.261795+0000","last_undegraded":"2023-12-16T21:57:45.261795+0000","last_fullsized":"2023-12-16T21:57:45.261795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457982+0000","last_change":"2023-12-16T21:57:43.494064+0000","last_active":"2023-12-16T21:57:44.457982+0000","last_peered":"2023-12-16T21:57:44.457982+0000","last_clean":"2023-12-16T21:57:44.457982+0000","last_became_active":"2023-12-16T21:57:43.493374+0000","last_became_peered":"2023-12-16T21:57:43.493374+0000","last_unstale":"2023-12-16T21:57:44.457982+0000","last_undegraded":"2023-12-16T21:57:44.457982+0000","last_fullsized":"2023-12-16T21:57:44.457982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:09.494346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869828+0000","last_change":"2023-12-16T21:57:42.508931+0000","last_active":"2023-12-16T21:57:44.869828+0000","last_peered":"2023-12-16T21:57:44.869828+0000","last_clean":"2023-12-16T21:57:44.869828+0000","last_became_active":"2023-12-16T21:57:42.508438+0000","last_became_peered":"2023-12-16T21:57:42.508438+0000","last_unstale":"2023-12-16T21:57:44.869828+0000","last_undegraded":"2023-12-16T21:57:44.869828+0000","last_fullsized":"2023-12-16T21:57:44.869828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:39.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262488+0000","last_change":"2023-12-16T21:57:43.485012+0000","last_active":"2023-12-16T21:57:45.262488+0000","last_peered":"2023-12-16T21:57:45.262488+0000","last_clean":"2023-12-16T21:57:45.262488+0000","last_became_active":"2023-12-16T21:57:43.484177+0000","last_became_peered":"2023-12-16T21:57:43.484177+0000","last_unstale":"2023-12-16T21:57:45.262488+0000","last_undegraded":"2023-12-16T21:57:45.262488+0000","last_fullsized":"2023-12-16T21:57:45.262488+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869858+0000","last_change":"2023-12-16T21:57:42.504338+0000","last_active":"2023-12-16T21:57:44.869858+0000","last_peered":"2023-12-16T21:57:44.869858+0000","last_clean":"2023-12-16T21:57:44.869858+0000","last_became_active":"2023-12-16T21:57:42.503125+0000","last_became_peered":"2023-12-16T21:57:42.503125+0000","last_unstale":"2023-12-16T21:57:44.869858+0000","last_undegraded":"2023-12-16T21:57:44.869858+0000","last_fullsized":"2023-12-16T21:57:44.869858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:23:06.394772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262761+0000","last_change":"2023-12-16T21:57:43.499585+0000","last_active":"2023-12-16T21:57:45.262761+0000","last_peered":"2023-12-16T21:57:45.262761+0000","last_clean":"2023-12-16T21:57:45.262761+0000","last_became_active":"2023-12-16T21:57:43.498790+0000","last_became_peered":"2023-12-16T21:57:43.498790+0000","last_unstale":"2023-12-16T21:57:45.262761+0000","last_undegraded":"2023-12-16T21:57:45.262761+0000","last_fullsized":"2023-12-16T21:57:45.262761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264439+0000","last_change":"2023-12-16T21:57:42.522249+0000","last_active":"2023-12-16T21:57:45.264439+0000","last_peered":"2023-12-16T21:57:45.264439+0000","last_clean":"2023-12-16T21:57:45.264439+0000","last_became_active":"2023-12-16T21:57:42.521585+0000","last_became_peered":"2023-12-16T21:57:42.521585+0000","last_unstale":"2023-12-16T21:57:45.264439+0000","last_undegraded":"2023-12-16T21:57:45.264439+0000","last_fullsized":"2023-12-16T21:57:45.264439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:22.825415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457737+0000","last_change":"2023-12-16T21:57:43.495929+0000","last_active":"2023-12-16T21:57:44.457737+0000","last_peered":"2023-12-16T21:57:44.457737+0000","last_clean":"2023-12-16T21:57:44.457737+0000","last_became_active":"2023-12-16T21:57:43.494696+0000","last_became_peered":"2023-12-16T21:57:43.494696+0000","last_unstale":"2023-12-16T21:57:44.457737+0000","last_undegraded":"2023-12-16T21:57:44.457737+0000","last_fullsized":"2023-12-16T21:57:44.457737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262280+0000","last_change":"2023-12-16T21:57:43.496545+0000","last_active":"2023-12-16T21:57:45.262280+0000","last_peered":"2023-12-16T21:57:45.262280+0000","last_clean":"2023-12-16T21:57:45.262280+0000","last_became_active":"2023-12-16T21:57:43.495846+0000","last_became_peered":"2023-12-16T21:57:43.495846+0000","last_unstale":"2023-12-16T21:57:45.262280+0000","last_undegraded":"2023-12-16T21:57:45.262280+0000","last_fullsized":"2023-12-16T21:57:45.262280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264212+0000","last_change":"2023-12-16T21:57:42.524077+0000","last_active":"2023-12-16T21:57:45.264212+0000","last_peered":"2023-12-16T21:57:45.264212+0000","last_clean":"2023-12-16T21:57:45.264212+0000","last_became_active":"2023-12-16T21:57:42.523590+0000","last_became_peered":"2023-12-16T21:57:42.523590+0000","last_unstale":"2023-12-16T21:57:45.264212+0000","last_undegraded":"2023-12-16T21:57:45.264212+0000","last_fullsized":"2023-12-16T21:57:45.264212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:57.160385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262278+0000","last_change":"2023-12-16T21:57:43.508947+0000","last_active":"2023-12-16T21:57:45.262278+0000","last_peered":"2023-12-16T21:57:45.262278+0000","last_clean":"2023-12-16T21:57:45.262278+0000","last_became_active":"2023-12-16T21:57:43.508424+0000","last_became_peered":"2023-12-16T21:57:43.508424+0000","last_unstale":"2023-12-16T21:57:45.262278+0000","last_undegraded":"2023-12-16T21:57:45.262278+0000","last_fullsized":"2023-12-16T21:57:45.262278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264760+0000","last_change":"2023-12-16T21:57:42.522477+0000","last_active":"2023-12-16T21:57:45.264760+0000","last_peered":"2023-12-16T21:57:45.264760+0000","last_clean":"2023-12-16T21:57:45.264760+0000","last_became_active":"2023-12-16T21:57:42.521732+0000","last_became_peered":"2023-12-16T21:57:42.521732+0000","last_unstale":"2023-12-16T21:57:45.264760+0000","last_undegraded":"2023-12-16T21:57:45.264760+0000","last_fullsized":"2023-12-16T21:57:45.264760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252926+0000","last_change":"2023-12-16T21:57:43.481470+0000","last_active":"2023-12-16T21:57:45.252926+0000","last_peered":"2023-12-16T21:57:45.252926+0000","last_clean":"2023-12-16T21:57:45.252926+0000","last_became_active":"2023-12-16T21:57:43.480541+0000","last_became_peered":"2023-12-16T21:57:43.480541+0000","last_unstale":"2023-12-16T21:57:45.252926+0000","last_undegraded":"2023-12-16T21:57:45.252926+0000","last_fullsized":"2023-12-16T21:57:45.252926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263338+0000","last_change":"2023-12-16T21:57:42.512776+0000","last_active":"2023-12-16T21:57:45.263338+0000","last_peered":"2023-12-16T21:57:45.263338+0000","last_clean":"2023-12-16T21:57:45.263338+0000","last_became_active":"2023-12-16T21:57:42.512393+0000","last_became_peered":"2023-12-16T21:57:42.512393+0000","last_unstale":"2023-12-16T21:57:45.263338+0000","last_undegraded":"2023-12-16T21:57:45.263338+0000","last_fullsized":"2023-12-16T21:57:45.263338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263375+0000","last_change":"2023-12-16T21:57:43.499571+0000","last_active":"2023-12-16T21:57:45.263375+0000","last_peered":"2023-12-16T21:57:45.263375+0000","last_clean":"2023-12-16T21:57:45.263375+0000","last_became_active":"2023-12-16T21:57:43.498880+0000","last_became_peered":"2023-12-16T21:57:43.498880+0000","last_unstale":"2023-12-16T21:57:45.263375+0000","last_undegraded":"2023-12-16T21:57:45.263375+0000","last_fullsized":"2023-12-16T21:57:45.263375+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.253109+0000","last_change":"2023-12-16T21:57:42.522619+0000","last_active":"2023-12-16T21:57:45.253109+0000","last_peered":"2023-12-16T21:57:45.253109+0000","last_clean":"2023-12-16T21:57:45.253109+0000","last_became_active":"2023-12-16T21:57:42.522259+0000","last_became_peered":"2023-12-16T21:57:42.522259+0000","last_unstale":"2023-12-16T21:57:45.253109+0000","last_undegraded":"2023-12-16T21:57:45.253109+0000","last_fullsized":"2023-12-16T21:57:45.253109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476743,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":827392,"data_stored":670753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]}]},{"osd":6,"up_from":16,"seq":68719476742,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]}]},{"osd":5,"up_from":16,"seq":68719476743,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":79893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]}]},{"osd":4,"up_from":16,"seq":68719476741,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":79929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":241664,"data_stored":81415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":237568,"data_stored":79963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.363}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":670809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":79803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:57:50.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:57:50.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa77ca640 1 Processor -- start 2023-12-16T21:57:50.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa77ca640 1 -- start start 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa77ca640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1aa015aee0 0x7f1aa015f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa77ca640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa015b2c0 0x7f1aa006a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa77ca640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1aa00ce6f0 0x7f1aa0070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa77ca640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa00aee80 con 0x7f1aa00ce6f0 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa77ca640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa015fbe0 con 0x7f1aa015aee0 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa77ca640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00cec40 con 0x7f1aa015b2c0 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1aa00ce6f0 0x7f1aa0070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1aa00ce6f0 0x7f1aa0070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46170/0 (socket says 172.21.15.28:46170) 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 -- 172.21.15.28:0/293404065 learned_addr learned my addr 172.21.15.28:0/293404065 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5566640 1 --2- 172.21.15.28:0/293404065 >> v2:172.21.15.122:3300/0 conn(0x7f1aa015aee0 0x7f1aa015f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa4d65640 1 --2- 172.21.15.28:0/293404065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa015b2c0 0x7f1aa006a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 -- 172.21.15.28:0/293404065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa015b2c0 msgr2=0x7f1aa006a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 --2- 172.21.15.28:0/293404065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa015b2c0 0x7f1aa006a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 -- 172.21.15.28:0/293404065 >> v2:172.21.15.122:3300/0 conn(0x7f1aa015aee0 msgr2=0x7f1aa015f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 --2- 172.21.15.28:0/293404065 >> v2:172.21.15.122:3300/0 conn(0x7f1aa015aee0 0x7f1aa015f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 -- 172.21.15.28:0/293404065 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00cef20 con 0x7f1aa00ce6f0 2023-12-16T21:57:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa4d65640 1 --2- 172.21.15.28:0/293404065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa015b2c0 0x7f1aa006a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1aa5d67640 1 --2- 172.21.15.28:0/293404065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa00ce6f0 0x7f1aa0070cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00d03b0 tx=0x7f1a9801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fefc0594af08c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1a97fff640 1 -- 172.21.15.28:0/293404065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980a56a0 con 0x7f1aa00ce6f0 2023-12-16T21:57:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1a97fff640 1 -- 172.21.15.28:0/293404065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a980a5800 con 0x7f1aa00ce6f0 2023-12-16T21:57:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.356+0000 7f1a97fff640 1 -- 172.21.15.28:0/293404065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980a5b90 con 0x7f1aa00ce6f0 2023-12-16T21:57:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- 172.21.15.28:0/293404065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa00ce6f0 msgr2=0x7f1aa0070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/293404065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa00ce6f0 0x7f1aa0070cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00d03b0 tx=0x7f1a9801c910 comp rx=0 tx=0).stop 2023-12-16T21:57:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- 172.21.15.28:0/293404065 shutdown_connections 2023-12-16T21:57:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/293404065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa00ce6f0 0x7f1aa0070cb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/293404065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa015b2c0 0x7f1aa006a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/293404065 >> v2:172.21.15.122:3300/0 conn(0x7f1aa015aee0 0x7f1aa015f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- 172.21.15.28:0/293404065 >> 172.21.15.28:0/293404065 conn(0x7f1aa00cf5d0 msgr2=0x7f1aa00cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- 172.21.15.28:0/293404065 shutdown_connections 2023-12-16T21:57:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- 172.21.15.28:0/293404065 wait complete. 2023-12-16T21:57:50.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 Processor -- start 2023-12-16T21:57:50.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- start start 2023-12-16T21:57:50.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa00ce6f0 0x7f1aa01493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa015aee0 0x7f1aa015df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa015b2c0 0x7f1aa014c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0149b40 con 0x7f1aa015aee0 2023-12-16T21:57:50.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0149d90 con 0x7f1aa015b2c0 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa014a070 con 0x7f1aa00ce6f0 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa015aee0 0x7f1aa015df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa015aee0 0x7f1aa015df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46180/0 (socket says 172.21.15.28:46180) 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 -- 172.21.15.28:0/670126898 learned_addr learned my addr 172.21.15.28:0/670126898 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa5d67640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa015b2c0 0x7f1aa014c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa5566640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa00ce6f0 0x7f1aa01493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 -- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa00ce6f0 msgr2=0x7f1aa01493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa00ce6f0 0x7f1aa01493a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 -- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa015b2c0 msgr2=0x7f1aa014c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa015b2c0 0x7f1aa014c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 -- 172.21.15.28:0/670126898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a98009b30 con 0x7f1aa015aee0 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa4d65640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa015aee0 0x7f1aa015df90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000c7f0 tx=0x7f1a9000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1a95ffb640 1 -- 172.21.15.28:0/670126898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9000cea0 con 0x7f1aa015aee0 2023-12-16T21:57:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1a95ffb640 1 -- 172.21.15.28:0/670126898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a90007a30 con 0x7f1aa015aee0 2023-12-16T21:57:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa014a350 con 0x7f1aa015aee0 2023-12-16T21:57:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1a95ffb640 1 -- 172.21.15.28:0/670126898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90007dc0 con 0x7f1aa015aee0 2023-12-16T21:57:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa014a910 con 0x7f1aa015aee0 2023-12-16T21:57:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.360+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa014ab30 con 0x7f1aa015aee0 2023-12-16T21:57:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.364+0000 7f1a95ffb640 1 -- 172.21.15.28:0/670126898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1a900113f0 con 0x7f1aa015aee0 2023-12-16T21:57:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.364+0000 7f1a95ffb640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a84066470 0x7f1a84068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.364+0000 7f1a95ffb640 1 -- 172.21.15.28:0/670126898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a90083fb0 con 0x7f1aa015aee0 2023-12-16T21:57:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.364+0000 7f1aa5566640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a84066470 0x7f1a84068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.364+0000 7f1aa5566640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a84066470 0x7f1a84068930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0154b90 tx=0x7f1a9c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.368+0000 7f1a95ffb640 1 -- 172.21.15.28:0/670126898 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f1a900b5290 con 0x7f1aa015aee0 2023-12-16T21:57:50.515 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.516+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1aa014a1e0 con 0x7f1a84066470 2023-12-16T21:57:50.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.532+0000 7f1a95ffb640 1 -- 172.21.15.28:0/670126898 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264601 (secure 0 0 0) 0x7f1aa014a1e0 con 0x7f1a84066470 2023-12-16T21:57:50.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:57:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a84066470 msgr2=0x7f1a84068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a84066470 0x7f1a84068930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0154b90 tx=0x7f1a9c008040 comp rx=0 tx=0).stop 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa015aee0 msgr2=0x7f1aa015df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa015aee0 0x7f1aa015df90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000c7f0 tx=0x7f1a9000ccb0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 shutdown_connections 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a84066470 0x7f1a84068930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa015b2c0 0x7f1aa014c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa015aee0 0x7f1aa015df90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 --2- 172.21.15.28:0/670126898 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa00ce6f0 0x7f1aa01493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 >> 172.21.15.28:0/670126898 conn(0x7f1aa00cf5d0 msgr2=0x7f1aa00b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 shutdown_connections 2023-12-16T21:57:50.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.536+0000 7f1aa77ca640 1 -- 172.21.15.28:0/670126898 wait complete. 2023-12-16T21:57:50.553 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-12-16T21:57:49.383589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235824,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738896,"statfs":{"total":773094113280,"available":772852629504,"internally_reserved":0,"allocated":3088384,"data_stored":1823980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.906281"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264489+0000","last_change":"2023-12-16T21:57:43.518611+0000","last_active":"2023-12-16T21:57:45.264489+0000","last_peered":"2023-12-16T21:57:45.264489+0000","last_clean":"2023-12-16T21:57:45.264489+0000","last_became_active":"2023-12-16T21:57:43.517704+0000","last_became_peered":"2023-12-16T21:57:43.517704+0000","last_unstale":"2023-12-16T21:57:45.264489+0000","last_undegraded":"2023-12-16T21:57:45.264489+0000","last_fullsized":"2023-12-16T21:57:45.264489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:49:13.751852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870153+0000","last_change":"2023-12-16T21:57:43.485837+0000","last_active":"2023-12-16T21:57:44.870153+0000","last_peered":"2023-12-16T21:57:44.870153+0000","last_clean":"2023-12-16T21:57:44.870153+0000","last_became_active":"2023-12-16T21:57:43.484881+0000","last_became_peered":"2023-12-16T21:57:43.484881+0000","last_unstale":"2023-12-16T21:57:44.870153+0000","last_undegraded":"2023-12-16T21:57:44.870153+0000","last_fullsized":"2023-12-16T21:57:44.870153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252805+0000","last_change":"2023-12-16T21:57:43.492016+0000","last_active":"2023-12-16T21:57:45.252805+0000","last_peered":"2023-12-16T21:57:45.252805+0000","last_clean":"2023-12-16T21:57:45.252805+0000","last_became_active":"2023-12-16T21:57:43.491114+0000","last_became_peered":"2023-12-16T21:57:43.491114+0000","last_unstale":"2023-12-16T21:57:45.252805+0000","last_undegraded":"2023-12-16T21:57:45.252805+0000","last_fullsized":"2023-12-16T21:57:45.252805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262086+0000","last_change":"2023-12-16T21:57:43.510356+0000","last_active":"2023-12-16T21:57:45.262086+0000","last_peered":"2023-12-16T21:57:45.262086+0000","last_clean":"2023-12-16T21:57:45.262086+0000","last_became_active":"2023-12-16T21:57:43.509819+0000","last_became_peered":"2023-12-16T21:57:43.509819+0000","last_unstale":"2023-12-16T21:57:45.262086+0000","last_undegraded":"2023-12-16T21:57:45.262086+0000","last_fullsized":"2023-12-16T21:57:45.262086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264164+0000","last_change":"2023-12-16T21:57:43.507126+0000","last_active":"2023-12-16T21:57:45.264164+0000","last_peered":"2023-12-16T21:57:45.264164+0000","last_clean":"2023-12-16T21:57:45.264164+0000","last_became_active":"2023-12-16T21:57:43.506352+0000","last_became_peered":"2023-12-16T21:57:43.506352+0000","last_unstale":"2023-12-16T21:57:45.264164+0000","last_undegraded":"2023-12-16T21:57:45.264164+0000","last_fullsized":"2023-12-16T21:57:45.264164+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:04:09.033093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262865+0000","last_change":"2023-12-16T21:57:43.483088+0000","last_active":"2023-12-16T21:57:45.262865+0000","last_peered":"2023-12-16T21:57:45.262865+0000","last_clean":"2023-12-16T21:57:45.262865+0000","last_became_active":"2023-12-16T21:57:43.482278+0000","last_became_peered":"2023-12-16T21:57:43.482278+0000","last_unstale":"2023-12-16T21:57:45.262865+0000","last_undegraded":"2023-12-16T21:57:45.262865+0000","last_fullsized":"2023-12-16T21:57:45.262865+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:15.360923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261922+0000","last_change":"2023-12-16T21:57:43.494598+0000","last_active":"2023-12-16T21:57:45.261922+0000","last_peered":"2023-12-16T21:57:45.261922+0000","last_clean":"2023-12-16T21:57:45.261922+0000","last_became_active":"2023-12-16T21:57:43.493867+0000","last_became_peered":"2023-12-16T21:57:43.493867+0000","last_unstale":"2023-12-16T21:57:45.261922+0000","last_undegraded":"2023-12-16T21:57:45.261922+0000","last_fullsized":"2023-12-16T21:57:45.261922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262085+0000","last_change":"2023-12-16T21:57:43.504864+0000","last_active":"2023-12-16T21:57:45.262085+0000","last_peered":"2023-12-16T21:57:45.262085+0000","last_clean":"2023-12-16T21:57:45.262085+0000","last_became_active":"2023-12-16T21:57:43.504140+0000","last_became_peered":"2023-12-16T21:57:43.504140+0000","last_unstale":"2023-12-16T21:57:45.262085+0000","last_undegraded":"2023-12-16T21:57:45.262085+0000","last_fullsized":"2023-12-16T21:57:45.262085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:22.190906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264734+0000","last_change":"2023-12-16T21:57:43.518225+0000","last_active":"2023-12-16T21:57:45.264734+0000","last_peered":"2023-12-16T21:57:45.264734+0000","last_clean":"2023-12-16T21:57:45.264734+0000","last_became_active":"2023-12-16T21:57:43.517602+0000","last_became_peered":"2023-12-16T21:57:43.517602+0000","last_unstale":"2023-12-16T21:57:45.264734+0000","last_undegraded":"2023-12-16T21:57:45.264734+0000","last_fullsized":"2023-12-16T21:57:45.264734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:46:04.514997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.460177+0000","last_change":"2023-12-16T21:57:43.468564+0000","last_active":"2023-12-16T21:57:44.460177+0000","last_peered":"2023-12-16T21:57:44.460177+0000","last_clean":"2023-12-16T21:57:44.460177+0000","last_became_active":"2023-12-16T21:57:43.467674+0000","last_became_peered":"2023-12-16T21:57:43.467674+0000","last_unstale":"2023-12-16T21:57:44.460177+0000","last_undegraded":"2023-12-16T21:57:44.460177+0000","last_fullsized":"2023-12-16T21:57:44.460177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458605+0000","last_change":"2023-12-16T21:57:43.474153+0000","last_active":"2023-12-16T21:57:44.458605+0000","last_peered":"2023-12-16T21:57:44.458605+0000","last_clean":"2023-12-16T21:57:44.458605+0000","last_became_active":"2023-12-16T21:57:43.467788+0000","last_became_peered":"2023-12-16T21:57:43.467788+0000","last_unstale":"2023-12-16T21:57:44.458605+0000","last_undegraded":"2023-12-16T21:57:44.458605+0000","last_fullsized":"2023-12-16T21:57:44.458605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458900+0000","last_change":"2023-12-16T21:57:43.494364+0000","last_active":"2023-12-16T21:57:44.458900+0000","last_peered":"2023-12-16T21:57:44.458900+0000","last_clean":"2023-12-16T21:57:44.458900+0000","last_became_active":"2023-12-16T21:57:43.491952+0000","last_became_peered":"2023-12-16T21:57:43.491952+0000","last_unstale":"2023-12-16T21:57:44.458900+0000","last_undegraded":"2023-12-16T21:57:44.458900+0000","last_fullsized":"2023-12-16T21:57:44.458900+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:16.932172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262657+0000","last_change":"2023-12-16T21:57:43.490348+0000","last_active":"2023-12-16T21:57:45.262657+0000","last_peered":"2023-12-16T21:57:45.262657+0000","last_clean":"2023-12-16T21:57:45.262657+0000","last_became_active":"2023-12-16T21:57:43.489582+0000","last_became_peered":"2023-12-16T21:57:43.489582+0000","last_unstale":"2023-12-16T21:57:45.262657+0000","last_undegraded":"2023-12-16T21:57:45.262657+0000","last_fullsized":"2023-12-16T21:57:45.262657+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252238+0000","last_change":"2023-12-16T21:57:43.481446+0000","last_active":"2023-12-16T21:57:45.252238+0000","last_peered":"2023-12-16T21:57:45.252238+0000","last_clean":"2023-12-16T21:57:45.252238+0000","last_became_active":"2023-12-16T21:57:43.480422+0000","last_became_peered":"2023-12-16T21:57:43.480422+0000","last_unstale":"2023-12-16T21:57:45.252238+0000","last_undegraded":"2023-12-16T21:57:45.252238+0000","last_fullsized":"2023-12-16T21:57:45.252238+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869559+0000","last_change":"2023-12-16T21:57:43.507792+0000","last_active":"2023-12-16T21:57:44.869559+0000","last_peered":"2023-12-16T21:57:44.869559+0000","last_clean":"2023-12-16T21:57:44.869559+0000","last_became_active":"2023-12-16T21:57:43.507030+0000","last_became_peered":"2023-12-16T21:57:43.507030+0000","last_unstale":"2023-12-16T21:57:44.869559+0000","last_undegraded":"2023-12-16T21:57:44.869559+0000","last_fullsized":"2023-12-16T21:57:44.869559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261659+0000","last_change":"2023-12-16T21:57:43.485531+0000","last_active":"2023-12-16T21:57:45.261659+0000","last_peered":"2023-12-16T21:57:45.261659+0000","last_clean":"2023-12-16T21:57:45.261659+0000","last_became_active":"2023-12-16T21:57:43.482538+0000","last_became_peered":"2023-12-16T21:57:43.482538+0000","last_unstale":"2023-12-16T21:57:45.261659+0000","last_undegraded":"2023-12-16T21:57:45.261659+0000","last_fullsized":"2023-12-16T21:57:45.261659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263056+0000","last_change":"2023-12-16T21:57:43.507572+0000","last_active":"2023-12-16T21:57:45.263056+0000","last_peered":"2023-12-16T21:57:45.263056+0000","last_clean":"2023-12-16T21:57:45.263056+0000","last_became_active":"2023-12-16T21:57:43.507072+0000","last_became_peered":"2023-12-16T21:57:43.507072+0000","last_unstale":"2023-12-16T21:57:45.263056+0000","last_undegraded":"2023-12-16T21:57:45.263056+0000","last_fullsized":"2023-12-16T21:57:45.263056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:49:13.751852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.456131+0000","last_change":"2023-12-16T21:57:43.487657+0000","last_active":"2023-12-16T21:57:44.456131+0000","last_peered":"2023-12-16T21:57:44.456131+0000","last_clean":"2023-12-16T21:57:44.456131+0000","last_became_active":"2023-12-16T21:57:43.484848+0000","last_became_peered":"2023-12-16T21:57:43.484848+0000","last_unstale":"2023-12-16T21:57:44.456131+0000","last_undegraded":"2023-12-16T21:57:44.456131+0000","last_fullsized":"2023-12-16T21:57:44.456131+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:14:59.932149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264990+0000","last_change":"2023-12-16T21:57:43.503136+0000","last_active":"2023-12-16T21:57:45.264990+0000","last_peered":"2023-12-16T21:57:45.264990+0000","last_clean":"2023-12-16T21:57:45.264990+0000","last_became_active":"2023-12-16T21:57:43.502321+0000","last_became_peered":"2023-12-16T21:57:43.502321+0000","last_unstale":"2023-12-16T21:57:45.264990+0000","last_undegraded":"2023-12-16T21:57:45.264990+0000","last_fullsized":"2023-12-16T21:57:45.264990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:29.457727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458117+0000","last_change":"2023-12-16T21:57:43.496151+0000","last_active":"2023-12-16T21:57:44.458117+0000","last_peered":"2023-12-16T21:57:44.458117+0000","last_clean":"2023-12-16T21:57:44.458117+0000","last_became_active":"2023-12-16T21:57:43.495293+0000","last_became_peered":"2023-12-16T21:57:43.495293+0000","last_unstale":"2023-12-16T21:57:44.458117+0000","last_undegraded":"2023-12-16T21:57:44.458117+0000","last_fullsized":"2023-12-16T21:57:44.458117+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252271+0000","last_change":"2023-12-16T21:57:43.507839+0000","last_active":"2023-12-16T21:57:45.252271+0000","last_peered":"2023-12-16T21:57:45.252271+0000","last_clean":"2023-12-16T21:57:45.252271+0000","last_became_active":"2023-12-16T21:57:43.507150+0000","last_became_peered":"2023-12-16T21:57:43.507150+0000","last_unstale":"2023-12-16T21:57:45.252271+0000","last_undegraded":"2023-12-16T21:57:45.252271+0000","last_fullsized":"2023-12-16T21:57:45.252271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263163+0000","last_change":"2023-12-16T21:57:43.487735+0000","last_active":"2023-12-16T21:57:45.263163+0000","last_peered":"2023-12-16T21:57:45.263163+0000","last_clean":"2023-12-16T21:57:45.263163+0000","last_became_active":"2023-12-16T21:57:43.486972+0000","last_became_peered":"2023-12-16T21:57:43.486972+0000","last_unstale":"2023-12-16T21:57:45.263163+0000","last_undegraded":"2023-12-16T21:57:45.263163+0000","last_fullsized":"2023-12-16T21:57:45.263163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:46:04.514997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262348+0000","last_change":"2023-12-16T21:57:43.513648+0000","last_active":"2023-12-16T21:57:45.262348+0000","last_peered":"2023-12-16T21:57:45.262348+0000","last_clean":"2023-12-16T21:57:45.262348+0000","last_became_active":"2023-12-16T21:57:43.513086+0000","last_became_peered":"2023-12-16T21:57:43.513086+0000","last_unstale":"2023-12-16T21:57:45.262348+0000","last_undegraded":"2023-12-16T21:57:45.262348+0000","last_fullsized":"2023-12-16T21:57:45.262348+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261798+0000","last_change":"2023-12-16T21:57:43.505290+0000","last_active":"2023-12-16T21:57:45.261798+0000","last_peered":"2023-12-16T21:57:45.261798+0000","last_clean":"2023-12-16T21:57:45.261798+0000","last_became_active":"2023-12-16T21:57:43.504600+0000","last_became_peered":"2023-12-16T21:57:43.504600+0000","last_unstale":"2023-12-16T21:57:45.261798+0000","last_undegraded":"2023-12-16T21:57:45.261798+0000","last_fullsized":"2023-12-16T21:57:45.261798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263041+0000","last_change":"2023-12-16T21:57:43.510116+0000","last_active":"2023-12-16T21:57:45.263041+0000","last_peered":"2023-12-16T21:57:45.263041+0000","last_clean":"2023-12-16T21:57:45.263041+0000","last_became_active":"2023-12-16T21:57:43.509260+0000","last_became_peered":"2023-12-16T21:57:43.509260+0000","last_unstale":"2023-12-16T21:57:45.263041+0000","last_undegraded":"2023-12-16T21:57:45.263041+0000","last_fullsized":"2023-12-16T21:57:45.263041+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.460344+0000","last_change":"2023-12-16T21:57:43.489237+0000","last_active":"2023-12-16T21:57:44.460344+0000","last_peered":"2023-12-16T21:57:44.460344+0000","last_clean":"2023-12-16T21:57:44.460344+0000","last_became_active":"2023-12-16T21:57:43.488252+0000","last_became_peered":"2023-12-16T21:57:43.488252+0000","last_unstale":"2023-12-16T21:57:44.460344+0000","last_undegraded":"2023-12-16T21:57:44.460344+0000","last_fullsized":"2023-12-16T21:57:44.460344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869771+0000","last_change":"2023-12-16T21:57:43.502339+0000","last_active":"2023-12-16T21:57:44.869771+0000","last_peered":"2023-12-16T21:57:44.869771+0000","last_clean":"2023-12-16T21:57:44.869771+0000","last_became_active":"2023-12-16T21:57:43.501434+0000","last_became_peered":"2023-12-16T21:57:43.501434+0000","last_unstale":"2023-12-16T21:57:44.869771+0000","last_undegraded":"2023-12-16T21:57:44.869771+0000","last_fullsized":"2023-12-16T21:57:44.869771+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263170+0000","last_change":"2023-12-16T21:57:43.509617+0000","last_active":"2023-12-16T21:57:45.263170+0000","last_peered":"2023-12-16T21:57:45.263170+0000","last_clean":"2023-12-16T21:57:45.263170+0000","last_became_active":"2023-12-16T21:57:43.508922+0000","last_became_peered":"2023-12-16T21:57:43.508922+0000","last_unstale":"2023-12-16T21:57:45.263170+0000","last_undegraded":"2023-12-16T21:57:45.263170+0000","last_fullsized":"2023-12-16T21:57:45.263170+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252408+0000","last_change":"2023-12-16T21:57:43.505933+0000","last_active":"2023-12-16T21:57:45.252408+0000","last_peered":"2023-12-16T21:57:45.252408+0000","last_clean":"2023-12-16T21:57:45.252408+0000","last_became_active":"2023-12-16T21:57:43.504463+0000","last_became_peered":"2023-12-16T21:57:43.504463+0000","last_unstale":"2023-12-16T21:57:45.252408+0000","last_undegraded":"2023-12-16T21:57:45.252408+0000","last_fullsized":"2023-12-16T21:57:45.252408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263332+0000","last_change":"2023-12-16T21:57:43.501738+0000","last_active":"2023-12-16T21:57:45.263332+0000","last_peered":"2023-12-16T21:57:45.263332+0000","last_clean":"2023-12-16T21:57:45.263332+0000","last_became_active":"2023-12-16T21:57:43.501014+0000","last_became_peered":"2023-12-16T21:57:43.501014+0000","last_unstale":"2023-12-16T21:57:45.263332+0000","last_undegraded":"2023-12-16T21:57:45.263332+0000","last_fullsized":"2023-12-16T21:57:45.263332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:47:38.932092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.456015+0000","last_change":"2023-12-16T21:57:43.477348+0000","last_active":"2023-12-16T21:57:44.456015+0000","last_peered":"2023-12-16T21:57:44.456015+0000","last_clean":"2023-12-16T21:57:44.456015+0000","last_became_active":"2023-12-16T21:57:43.476616+0000","last_became_peered":"2023-12-16T21:57:43.476616+0000","last_unstale":"2023-12-16T21:57:44.456015+0000","last_undegraded":"2023-12-16T21:57:44.456015+0000","last_fullsized":"2023-12-16T21:57:44.456015+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:18:16.866270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457542+0000","last_change":"2023-12-16T21:57:43.483725+0000","last_active":"2023-12-16T21:57:44.457542+0000","last_peered":"2023-12-16T21:57:44.457542+0000","last_clean":"2023-12-16T21:57:44.457542+0000","last_became_active":"2023-12-16T21:57:43.482870+0000","last_became_peered":"2023-12-16T21:57:43.482870+0000","last_unstale":"2023-12-16T21:57:44.457542+0000","last_undegraded":"2023-12-16T21:57:44.457542+0000","last_fullsized":"2023-12-16T21:57:44.457542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458048+0000","last_change":"2023-12-16T21:57:43.492281+0000","last_active":"2023-12-16T21:57:44.458048+0000","last_peered":"2023-12-16T21:57:44.458048+0000","last_clean":"2023-12-16T21:57:44.458048+0000","last_became_active":"2023-12-16T21:57:43.491538+0000","last_became_peered":"2023-12-16T21:57:43.491538+0000","last_unstale":"2023-12-16T21:57:44.458048+0000","last_undegraded":"2023-12-16T21:57:44.458048+0000","last_fullsized":"2023-12-16T21:57:44.458048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:21:57.846367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262966+0000","last_change":"2023-12-16T21:57:42.522074+0000","last_active":"2023-12-16T21:57:45.262966+0000","last_peered":"2023-12-16T21:57:45.262966+0000","last_clean":"2023-12-16T21:57:45.262966+0000","last_became_active":"2023-12-16T21:57:42.521769+0000","last_became_peered":"2023-12-16T21:57:42.521769+0000","last_unstale":"2023-12-16T21:57:45.262966+0000","last_undegraded":"2023-12-16T21:57:45.262966+0000","last_fullsized":"2023-12-16T21:57:45.262966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:42:59.000744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252714+0000","last_change":"2023-12-16T21:57:43.495840+0000","last_active":"2023-12-16T21:57:45.252714+0000","last_peered":"2023-12-16T21:57:45.252714+0000","last_clean":"2023-12-16T21:57:45.252714+0000","last_became_active":"2023-12-16T21:57:43.495114+0000","last_became_peered":"2023-12-16T21:57:43.495114+0000","last_unstale":"2023-12-16T21:57:45.252714+0000","last_undegraded":"2023-12-16T21:57:45.252714+0000","last_fullsized":"2023-12-16T21:57:45.252714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:16.434855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458001+0000","last_change":"2023-12-16T21:57:42.515225+0000","last_active":"2023-12-16T21:57:44.458001+0000","last_peered":"2023-12-16T21:57:44.458001+0000","last_clean":"2023-12-16T21:57:44.458001+0000","last_became_active":"2023-12-16T21:57:42.514509+0000","last_became_peered":"2023-12-16T21:57:42.514509+0000","last_unstale":"2023-12-16T21:57:44.458001+0000","last_undegraded":"2023-12-16T21:57:44.458001+0000","last_fullsized":"2023-12-16T21:57:44.458001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:21.179733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262272+0000","last_change":"2023-12-16T21:57:42.515658+0000","last_active":"2023-12-16T21:57:45.262272+0000","last_peered":"2023-12-16T21:57:45.262272+0000","last_clean":"2023-12-16T21:57:45.262272+0000","last_became_active":"2023-12-16T21:57:42.515275+0000","last_became_peered":"2023-12-16T21:57:42.515275+0000","last_unstale":"2023-12-16T21:57:45.262272+0000","last_undegraded":"2023-12-16T21:57:45.262272+0000","last_fullsized":"2023-12-16T21:57:45.262272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:00:29.774694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262293+0000","last_change":"2023-12-16T21:57:43.493987+0000","last_active":"2023-12-16T21:57:45.262293+0000","last_peered":"2023-12-16T21:57:45.262293+0000","last_clean":"2023-12-16T21:57:45.262293+0000","last_became_active":"2023-12-16T21:57:43.493193+0000","last_became_peered":"2023-12-16T21:57:43.493193+0000","last_unstale":"2023-12-16T21:57:45.262293+0000","last_undegraded":"2023-12-16T21:57:45.262293+0000","last_fullsized":"2023-12-16T21:57:45.262293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261893+0000","last_change":"2023-12-16T21:57:42.508909+0000","last_active":"2023-12-16T21:57:45.261893+0000","last_peered":"2023-12-16T21:57:45.261893+0000","last_clean":"2023-12-16T21:57:45.261893+0000","last_became_active":"2023-12-16T21:57:42.508250+0000","last_became_peered":"2023-12-16T21:57:42.508250+0000","last_unstale":"2023-12-16T21:57:45.261893+0000","last_undegraded":"2023-12-16T21:57:45.261893+0000","last_fullsized":"2023-12-16T21:57:45.261893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:39.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869537+0000","last_change":"2023-12-16T21:57:43.487695+0000","last_active":"2023-12-16T21:57:44.869537+0000","last_peered":"2023-12-16T21:57:44.869537+0000","last_clean":"2023-12-16T21:57:44.869537+0000","last_became_active":"2023-12-16T21:57:43.486936+0000","last_became_peered":"2023-12-16T21:57:43.486936+0000","last_unstale":"2023-12-16T21:57:44.869537+0000","last_undegraded":"2023-12-16T21:57:44.869537+0000","last_fullsized":"2023-12-16T21:57:44.869537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:45:28.851428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458471+0000","last_change":"2023-12-16T21:57:42.526210+0000","last_active":"2023-12-16T21:57:44.458471+0000","last_peered":"2023-12-16T21:57:44.458471+0000","last_clean":"2023-12-16T21:57:44.458471+0000","last_became_active":"2023-12-16T21:57:42.525460+0000","last_became_peered":"2023-12-16T21:57:42.525460+0000","last_unstale":"2023-12-16T21:57:44.458471+0000","last_undegraded":"2023-12-16T21:57:44.458471+0000","last_fullsized":"2023-12-16T21:57:44.458471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:43:36.373508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263137+0000","last_change":"2023-12-16T21:57:43.483521+0000","last_active":"2023-12-16T21:57:45.263137+0000","last_peered":"2023-12-16T21:57:45.263137+0000","last_clean":"2023-12-16T21:57:45.263137+0000","last_became_active":"2023-12-16T21:57:43.482802+0000","last_became_peered":"2023-12-16T21:57:43.482802+0000","last_unstale":"2023-12-16T21:57:45.263137+0000","last_undegraded":"2023-12-16T21:57:45.263137+0000","last_fullsized":"2023-12-16T21:57:45.263137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:29.175289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263361+0000","last_change":"2023-12-16T21:57:42.517029+0000","last_active":"2023-12-16T21:57:45.263361+0000","last_peered":"2023-12-16T21:57:45.263361+0000","last_clean":"2023-12-16T21:57:45.263361+0000","last_became_active":"2023-12-16T21:57:42.516592+0000","last_became_peered":"2023-12-16T21:57:42.516592+0000","last_unstale":"2023-12-16T21:57:45.263361+0000","last_undegraded":"2023-12-16T21:57:45.263361+0000","last_fullsized":"2023-12-16T21:57:45.263361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:42:59.000744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252990+0000","last_change":"2023-12-16T21:57:43.480932+0000","last_active":"2023-12-16T21:57:45.252990+0000","last_peered":"2023-12-16T21:57:45.252990+0000","last_clean":"2023-12-16T21:57:45.252990+0000","last_became_active":"2023-12-16T21:57:43.480056+0000","last_became_peered":"2023-12-16T21:57:43.480056+0000","last_unstale":"2023-12-16T21:57:45.252990+0000","last_undegraded":"2023-12-16T21:57:45.252990+0000","last_fullsized":"2023-12-16T21:57:45.252990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:07:42.685694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869739+0000","last_change":"2023-12-16T21:57:42.499117+0000","last_active":"2023-12-16T21:57:44.869739+0000","last_peered":"2023-12-16T21:57:44.869739+0000","last_clean":"2023-12-16T21:57:44.869739+0000","last_became_active":"2023-12-16T21:57:42.498315+0000","last_became_peered":"2023-12-16T21:57:42.498315+0000","last_unstale":"2023-12-16T21:57:44.869739+0000","last_undegraded":"2023-12-16T21:57:44.869739+0000","last_fullsized":"2023-12-16T21:57:44.869739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:14:36.842626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459102+0000","last_change":"2023-12-16T21:57:43.507811+0000","last_active":"2023-12-16T21:57:44.459102+0000","last_peered":"2023-12-16T21:57:44.459102+0000","last_clean":"2023-12-16T21:57:44.459102+0000","last_became_active":"2023-12-16T21:57:43.507089+0000","last_became_peered":"2023-12-16T21:57:43.507089+0000","last_unstale":"2023-12-16T21:57:44.459102+0000","last_undegraded":"2023-12-16T21:57:44.459102+0000","last_fullsized":"2023-12-16T21:57:44.459102+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:43:00.011918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458372+0000","last_change":"2023-12-16T21:57:42.512797+0000","last_active":"2023-12-16T21:57:44.458372+0000","last_peered":"2023-12-16T21:57:44.458372+0000","last_clean":"2023-12-16T21:57:44.458372+0000","last_became_active":"2023-12-16T21:57:42.512055+0000","last_became_peered":"2023-12-16T21:57:42.512055+0000","last_unstale":"2023-12-16T21:57:44.458372+0000","last_undegraded":"2023-12-16T21:57:44.458372+0000","last_fullsized":"2023-12-16T21:57:44.458372+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:15.920999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264337+0000","last_change":"2023-12-16T21:57:43.510475+0000","last_active":"2023-12-16T21:57:45.264337+0000","last_peered":"2023-12-16T21:57:45.264337+0000","last_clean":"2023-12-16T21:57:45.264337+0000","last_became_active":"2023-12-16T21:57:43.509720+0000","last_became_peered":"2023-12-16T21:57:43.509720+0000","last_unstale":"2023-12-16T21:57:45.264337+0000","last_undegraded":"2023-12-16T21:57:45.264337+0000","last_fullsized":"2023-12-16T21:57:45.264337+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:15.360923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252634+0000","last_change":"2023-12-16T21:57:42.518031+0000","last_active":"2023-12-16T21:57:45.252634+0000","last_peered":"2023-12-16T21:57:45.252634+0000","last_clean":"2023-12-16T21:57:45.252634+0000","last_became_active":"2023-12-16T21:57:42.517664+0000","last_became_peered":"2023-12-16T21:57:42.517664+0000","last_unstale":"2023-12-16T21:57:45.252634+0000","last_undegraded":"2023-12-16T21:57:45.252634+0000","last_fullsized":"2023-12-16T21:57:45.252634+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252215+0000","last_change":"2023-12-16T21:57:43.493896+0000","last_active":"2023-12-16T21:57:45.252215+0000","last_peered":"2023-12-16T21:57:45.252215+0000","last_clean":"2023-12-16T21:57:45.252215+0000","last_became_active":"2023-12-16T21:57:43.492917+0000","last_became_peered":"2023-12-16T21:57:43.492917+0000","last_unstale":"2023-12-16T21:57:45.252215+0000","last_undegraded":"2023-12-16T21:57:45.252215+0000","last_fullsized":"2023-12-16T21:57:45.252215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:28:22.190906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263439+0000","last_change":"2023-12-16T21:57:42.517161+0000","last_active":"2023-12-16T21:57:45.263439+0000","last_peered":"2023-12-16T21:57:45.263439+0000","last_clean":"2023-12-16T21:57:45.263439+0000","last_became_active":"2023-12-16T21:57:42.516388+0000","last_became_peered":"2023-12-16T21:57:42.516388+0000","last_unstale":"2023-12-16T21:57:45.263439+0000","last_undegraded":"2023-12-16T21:57:45.263439+0000","last_fullsized":"2023-12-16T21:57:45.263439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:30:15.423681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264304+0000","last_change":"2023-12-16T21:57:43.512923+0000","last_active":"2023-12-16T21:57:45.264304+0000","last_peered":"2023-12-16T21:57:45.264304+0000","last_clean":"2023-12-16T21:57:45.264304+0000","last_became_active":"2023-12-16T21:57:43.512254+0000","last_became_peered":"2023-12-16T21:57:43.512254+0000","last_unstale":"2023-12-16T21:57:45.264304+0000","last_undegraded":"2023-12-16T21:57:45.264304+0000","last_fullsized":"2023-12-16T21:57:45.264304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:20.834561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457916+0000","last_change":"2023-12-16T21:57:42.511480+0000","last_active":"2023-12-16T21:57:44.457916+0000","last_peered":"2023-12-16T21:57:44.457916+0000","last_clean":"2023-12-16T21:57:44.457916+0000","last_became_active":"2023-12-16T21:57:42.510772+0000","last_became_peered":"2023-12-16T21:57:42.510772+0000","last_unstale":"2023-12-16T21:57:44.457916+0000","last_undegraded":"2023-12-16T21:57:44.457916+0000","last_fullsized":"2023-12-16T21:57:44.457916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457878+0000","last_change":"2023-12-16T21:57:43.493903+0000","last_active":"2023-12-16T21:57:44.457878+0000","last_peered":"2023-12-16T21:57:44.457878+0000","last_clean":"2023-12-16T21:57:44.457878+0000","last_became_active":"2023-12-16T21:57:43.493169+0000","last_became_peered":"2023-12-16T21:57:43.493169+0000","last_unstale":"2023-12-16T21:57:44.457878+0000","last_undegraded":"2023-12-16T21:57:44.457878+0000","last_fullsized":"2023-12-16T21:57:44.457878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.401674+0000","last_change":"2023-12-16T21:57:42.498884+0000","last_active":"2023-12-16T21:57:44.401674+0000","last_peered":"2023-12-16T21:57:44.401674+0000","last_clean":"2023-12-16T21:57:44.401674+0000","last_became_active":"2023-12-16T21:57:42.498452+0000","last_became_peered":"2023-12-16T21:57:42.498452+0000","last_unstale":"2023-12-16T21:57:44.401674+0000","last_undegraded":"2023-12-16T21:57:44.401674+0000","last_fullsized":"2023-12-16T21:57:44.401674+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:37:03.298907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263857+0000","last_change":"2023-12-16T21:57:43.513792+0000","last_active":"2023-12-16T21:57:45.263857+0000","last_peered":"2023-12-16T21:57:45.263857+0000","last_clean":"2023-12-16T21:57:45.263857+0000","last_became_active":"2023-12-16T21:57:43.513001+0000","last_became_peered":"2023-12-16T21:57:43.513001+0000","last_unstale":"2023-12-16T21:57:45.263857+0000","last_undegraded":"2023-12-16T21:57:45.263857+0000","last_fullsized":"2023-12-16T21:57:45.263857+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459156+0000","last_change":"2023-12-16T21:57:42.512676+0000","last_active":"2023-12-16T21:57:44.459156+0000","last_peered":"2023-12-16T21:57:44.459156+0000","last_clean":"2023-12-16T21:57:44.459156+0000","last_became_active":"2023-12-16T21:57:42.512267+0000","last_became_peered":"2023-12-16T21:57:42.512267+0000","last_unstale":"2023-12-16T21:57:44.459156+0000","last_undegraded":"2023-12-16T21:57:44.459156+0000","last_fullsized":"2023-12-16T21:57:44.459156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:23:06.394772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264897+0000","last_change":"2023-12-16T21:57:43.517003+0000","last_active":"2023-12-16T21:57:45.264897+0000","last_peered":"2023-12-16T21:57:45.264897+0000","last_clean":"2023-12-16T21:57:45.264897+0000","last_became_active":"2023-12-16T21:57:43.516197+0000","last_became_peered":"2023-12-16T21:57:43.516197+0000","last_unstale":"2023-12-16T21:57:45.264897+0000","last_undegraded":"2023-12-16T21:57:45.264897+0000","last_fullsized":"2023-12-16T21:57:45.264897+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:48:39.071922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.401840+0000","last_change":"2023-12-16T21:57:42.512911+0000","last_active":"2023-12-16T21:57:44.401840+0000","last_peered":"2023-12-16T21:57:44.401840+0000","last_clean":"2023-12-16T21:57:44.401840+0000","last_became_active":"2023-12-16T21:57:42.512410+0000","last_became_peered":"2023-12-16T21:57:42.512410+0000","last_unstale":"2023-12-16T21:57:44.401840+0000","last_undegraded":"2023-12-16T21:57:44.401840+0000","last_fullsized":"2023-12-16T21:57:44.401840+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:00:58.952738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264838+0000","last_change":"2023-12-16T21:57:43.518693+0000","last_active":"2023-12-16T21:57:45.264838+0000","last_peered":"2023-12-16T21:57:45.264838+0000","last_clean":"2023-12-16T21:57:45.264838+0000","last_became_active":"2023-12-16T21:57:43.518184+0000","last_became_peered":"2023-12-16T21:57:43.518184+0000","last_unstale":"2023-12-16T21:57:45.264838+0000","last_undegraded":"2023-12-16T21:57:45.264838+0000","last_fullsized":"2023-12-16T21:57:45.264838+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:47:38.932092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869962+0000","last_change":"2023-12-16T21:57:42.505084+0000","last_active":"2023-12-16T21:57:44.869962+0000","last_peered":"2023-12-16T21:57:44.869962+0000","last_clean":"2023-12-16T21:57:44.869962+0000","last_became_active":"2023-12-16T21:57:42.502948+0000","last_became_peered":"2023-12-16T21:57:42.502948+0000","last_unstale":"2023-12-16T21:57:44.869962+0000","last_undegraded":"2023-12-16T21:57:44.869962+0000","last_fullsized":"2023-12-16T21:57:44.869962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262484+0000","last_change":"2023-12-16T21:57:43.499808+0000","last_active":"2023-12-16T21:57:45.262484+0000","last_peered":"2023-12-16T21:57:45.262484+0000","last_clean":"2023-12-16T21:57:45.262484+0000","last_became_active":"2023-12-16T21:57:43.498922+0000","last_became_peered":"2023-12-16T21:57:43.498922+0000","last_unstale":"2023-12-16T21:57:45.262484+0000","last_undegraded":"2023-12-16T21:57:45.262484+0000","last_fullsized":"2023-12-16T21:57:45.262484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:04:09.033093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.401759+0000","last_change":"2023-12-16T21:57:42.509838+0000","last_active":"2023-12-16T21:57:44.401759+0000","last_peered":"2023-12-16T21:57:44.401759+0000","last_clean":"2023-12-16T21:57:44.401759+0000","last_became_active":"2023-12-16T21:57:42.508994+0000","last_became_peered":"2023-12-16T21:57:42.508994+0000","last_unstale":"2023-12-16T21:57:44.401759+0000","last_undegraded":"2023-12-16T21:57:44.401759+0000","last_fullsized":"2023-12-16T21:57:44.401759+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:03:03.258824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264723+0000","last_change":"2023-12-16T21:57:43.516388+0000","last_active":"2023-12-16T21:57:45.264723+0000","last_peered":"2023-12-16T21:57:45.264723+0000","last_clean":"2023-12-16T21:57:45.264723+0000","last_became_active":"2023-12-16T21:57:43.515749+0000","last_became_peered":"2023-12-16T21:57:43.515749+0000","last_unstale":"2023-12-16T21:57:45.264723+0000","last_undegraded":"2023-12-16T21:57:45.264723+0000","last_fullsized":"2023-12-16T21:57:45.264723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:29.175289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:43.474023+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:57:43.474023+0000","last_peered":"2023-12-16T21:57:43.474023+0000","last_clean":"2023-12-16T21:57:43.474023+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:57:43.474023+0000","last_undegraded":"2023-12-16T21:57:43.474023+0000","last_fullsized":"2023-12-16T21:57:43.474023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261664+0000","last_change":"2023-12-16T21:57:42.508124+0000","last_active":"2023-12-16T21:57:45.261664+0000","last_peered":"2023-12-16T21:57:45.261664+0000","last_clean":"2023-12-16T21:57:45.261664+0000","last_became_active":"2023-12-16T21:57:42.507727+0000","last_became_peered":"2023-12-16T21:57:42.507727+0000","last_unstale":"2023-12-16T21:57:45.261664+0000","last_undegraded":"2023-12-16T21:57:45.261664+0000","last_fullsized":"2023-12-16T21:57:45.261664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262915+0000","last_change":"2023-12-16T21:57:43.490787+0000","last_active":"2023-12-16T21:57:45.262915+0000","last_peered":"2023-12-16T21:57:45.262915+0000","last_clean":"2023-12-16T21:57:45.262915+0000","last_became_active":"2023-12-16T21:57:43.490013+0000","last_became_peered":"2023-12-16T21:57:43.490013+0000","last_unstale":"2023-12-16T21:57:45.262915+0000","last_undegraded":"2023-12-16T21:57:45.262915+0000","last_fullsized":"2023-12-16T21:57:45.262915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262013+0000","last_change":"2023-12-16T21:57:42.523476+0000","last_active":"2023-12-16T21:57:45.262013+0000","last_peered":"2023-12-16T21:57:45.262013+0000","last_clean":"2023-12-16T21:57:45.262013+0000","last_became_active":"2023-12-16T21:57:42.522441+0000","last_became_peered":"2023-12-16T21:57:42.522441+0000","last_unstale":"2023-12-16T21:57:45.262013+0000","last_undegraded":"2023-12-16T21:57:45.262013+0000","last_fullsized":"2023-12-16T21:57:45.262013+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:17:15.920999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252089+0000","last_change":"2023-12-16T21:57:43.508913+0000","last_active":"2023-12-16T21:57:45.252089+0000","last_peered":"2023-12-16T21:57:45.252089+0000","last_clean":"2023-12-16T21:57:45.252089+0000","last_became_active":"2023-12-16T21:57:43.508133+0000","last_became_peered":"2023-12-16T21:57:43.508133+0000","last_unstale":"2023-12-16T21:57:45.252089+0000","last_undegraded":"2023-12-16T21:57:45.252089+0000","last_fullsized":"2023-12-16T21:57:45.252089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:58.171558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.401737+0000","last_change":"2023-12-16T21:57:42.513117+0000","last_active":"2023-12-16T21:57:44.401737+0000","last_peered":"2023-12-16T21:57:44.401737+0000","last_clean":"2023-12-16T21:57:44.401737+0000","last_became_active":"2023-12-16T21:57:42.512449+0000","last_became_peered":"2023-12-16T21:57:42.512449+0000","last_unstale":"2023-12-16T21:57:44.401737+0000","last_undegraded":"2023-12-16T21:57:44.401737+0000","last_fullsized":"2023-12-16T21:57:44.401737+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:55:25.298199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869573+0000","last_change":"2023-12-16T21:57:43.486971+0000","last_active":"2023-12-16T21:57:44.869573+0000","last_peered":"2023-12-16T21:57:44.869573+0000","last_clean":"2023-12-16T21:57:44.869573+0000","last_became_active":"2023-12-16T21:57:43.483854+0000","last_became_peered":"2023-12-16T21:57:43.483854+0000","last_unstale":"2023-12-16T21:57:44.869573+0000","last_undegraded":"2023-12-16T21:57:44.869573+0000","last_fullsized":"2023-12-16T21:57:44.869573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263608+0000","last_change":"2023-12-16T21:57:42.501969+0000","last_active":"2023-12-16T21:57:45.263608+0000","last_peered":"2023-12-16T21:57:45.263608+0000","last_clean":"2023-12-16T21:57:45.263608+0000","last_became_active":"2023-12-16T21:57:42.501059+0000","last_became_peered":"2023-12-16T21:57:42.501059+0000","last_unstale":"2023-12-16T21:57:45.263608+0000","last_undegraded":"2023-12-16T21:57:45.263608+0000","last_fullsized":"2023-12-16T21:57:45.263608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:09.342194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.459101+0000","last_change":"2023-12-16T21:57:43.506507+0000","last_active":"2023-12-16T21:57:44.459101+0000","last_peered":"2023-12-16T21:57:44.459101+0000","last_clean":"2023-12-16T21:57:44.459101+0000","last_became_active":"2023-12-16T21:57:43.505685+0000","last_became_peered":"2023-12-16T21:57:43.505685+0000","last_unstale":"2023-12-16T21:57:44.459101+0000","last_undegraded":"2023-12-16T21:57:44.459101+0000","last_fullsized":"2023-12-16T21:57:44.459101+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:40.013144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869952+0000","last_change":"2023-12-16T21:57:42.503616+0000","last_active":"2023-12-16T21:57:44.869952+0000","last_peered":"2023-12-16T21:57:44.869952+0000","last_clean":"2023-12-16T21:57:44.869952+0000","last_became_active":"2023-12-16T21:57:42.502114+0000","last_became_peered":"2023-12-16T21:57:42.502114+0000","last_unstale":"2023-12-16T21:57:44.869952+0000","last_undegraded":"2023-12-16T21:57:44.869952+0000","last_fullsized":"2023-12-16T21:57:44.869952+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:57.160385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262539+0000","last_change":"2023-12-16T21:57:43.500130+0000","last_active":"2023-12-16T21:57:45.262539+0000","last_peered":"2023-12-16T21:57:45.262539+0000","last_clean":"2023-12-16T21:57:45.262539+0000","last_became_active":"2023-12-16T21:57:43.499128+0000","last_became_peered":"2023-12-16T21:57:43.499128+0000","last_unstale":"2023-12-16T21:57:45.262539+0000","last_undegraded":"2023-12-16T21:57:45.262539+0000","last_fullsized":"2023-12-16T21:57:45.262539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:20.834561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.265035+0000","last_change":"2023-12-16T21:57:42.526337+0000","last_active":"2023-12-16T21:57:45.265035+0000","last_peered":"2023-12-16T21:57:45.265035+0000","last_clean":"2023-12-16T21:57:45.265035+0000","last_became_active":"2023-12-16T21:57:42.525796+0000","last_became_peered":"2023-12-16T21:57:42.525796+0000","last_unstale":"2023-12-16T21:57:45.265035+0000","last_undegraded":"2023-12-16T21:57:45.265035+0000","last_fullsized":"2023-12-16T21:57:45.265035+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:00:29.774694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.870185+0000","last_change":"2023-12-16T21:57:43.477509+0000","last_active":"2023-12-16T21:57:44.870185+0000","last_peered":"2023-12-16T21:57:44.870185+0000","last_clean":"2023-12-16T21:57:44.870185+0000","last_became_active":"2023-12-16T21:57:43.476854+0000","last_became_peered":"2023-12-16T21:57:43.476854+0000","last_unstale":"2023-12-16T21:57:44.870185+0000","last_undegraded":"2023-12-16T21:57:44.870185+0000","last_fullsized":"2023-12-16T21:57:44.870185+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.401085+0000","last_change":"2023-12-16T21:57:42.499802+0000","last_active":"2023-12-16T21:57:45.401085+0000","last_peered":"2023-12-16T21:57:45.401085+0000","last_clean":"2023-12-16T21:57:45.401085+0000","last_became_active":"2023-12-16T21:57:42.498817+0000","last_became_peered":"2023-12-16T21:57:42.498817+0000","last_unstale":"2023-12-16T21:57:45.401085+0000","last_undegraded":"2023-12-16T21:57:45.401085+0000","last_fullsized":"2023-12-16T21:57:45.401085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263239+0000","last_change":"2023-12-16T21:57:43.490695+0000","last_active":"2023-12-16T21:57:45.263239+0000","last_peered":"2023-12-16T21:57:45.263239+0000","last_clean":"2023-12-16T21:57:45.263239+0000","last_became_active":"2023-12-16T21:57:43.489927+0000","last_became_peered":"2023-12-16T21:57:43.489927+0000","last_unstale":"2023-12-16T21:57:45.263239+0000","last_undegraded":"2023-12-16T21:57:45.263239+0000","last_fullsized":"2023-12-16T21:57:45.263239+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.458559+0000","last_change":"2023-12-16T21:57:42.523142+0000","last_active":"2023-12-16T21:57:44.458559+0000","last_peered":"2023-12-16T21:57:44.458559+0000","last_clean":"2023-12-16T21:57:44.458559+0000","last_became_active":"2023-12-16T21:57:42.521597+0000","last_became_peered":"2023-12-16T21:57:42.521597+0000","last_unstale":"2023-12-16T21:57:44.458559+0000","last_undegraded":"2023-12-16T21:57:44.458559+0000","last_fullsized":"2023-12-16T21:57:44.458559+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252927+0000","last_change":"2023-12-16T21:57:43.483454+0000","last_active":"2023-12-16T21:57:45.252927+0000","last_peered":"2023-12-16T21:57:45.252927+0000","last_clean":"2023-12-16T21:57:45.252927+0000","last_became_active":"2023-12-16T21:57:43.482451+0000","last_became_peered":"2023-12-16T21:57:43.482451+0000","last_unstale":"2023-12-16T21:57:45.252927+0000","last_undegraded":"2023-12-16T21:57:45.252927+0000","last_fullsized":"2023-12-16T21:57:45.252927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:26:36.928082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.261795+0000","last_change":"2023-12-16T21:57:42.522091+0000","last_active":"2023-12-16T21:57:45.261795+0000","last_peered":"2023-12-16T21:57:45.261795+0000","last_clean":"2023-12-16T21:57:45.261795+0000","last_became_active":"2023-12-16T21:57:42.521328+0000","last_became_peered":"2023-12-16T21:57:42.521328+0000","last_unstale":"2023-12-16T21:57:45.261795+0000","last_undegraded":"2023-12-16T21:57:45.261795+0000","last_fullsized":"2023-12-16T21:57:45.261795+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:08.483173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457982+0000","last_change":"2023-12-16T21:57:43.494064+0000","last_active":"2023-12-16T21:57:44.457982+0000","last_peered":"2023-12-16T21:57:44.457982+0000","last_clean":"2023-12-16T21:57:44.457982+0000","last_became_active":"2023-12-16T21:57:43.493374+0000","last_became_peered":"2023-12-16T21:57:43.493374+0000","last_unstale":"2023-12-16T21:57:44.457982+0000","last_undegraded":"2023-12-16T21:57:44.457982+0000","last_fullsized":"2023-12-16T21:57:44.457982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:09.494346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869828+0000","last_change":"2023-12-16T21:57:42.508931+0000","last_active":"2023-12-16T21:57:44.869828+0000","last_peered":"2023-12-16T21:57:44.869828+0000","last_clean":"2023-12-16T21:57:44.869828+0000","last_became_active":"2023-12-16T21:57:42.508438+0000","last_became_peered":"2023-12-16T21:57:42.508438+0000","last_unstale":"2023-12-16T21:57:44.869828+0000","last_undegraded":"2023-12-16T21:57:44.869828+0000","last_fullsized":"2023-12-16T21:57:44.869828+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:39.001970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262488+0000","last_change":"2023-12-16T21:57:43.485012+0000","last_active":"2023-12-16T21:57:45.262488+0000","last_peered":"2023-12-16T21:57:45.262488+0000","last_clean":"2023-12-16T21:57:45.262488+0000","last_became_active":"2023-12-16T21:57:43.484177+0000","last_became_peered":"2023-12-16T21:57:43.484177+0000","last_unstale":"2023-12-16T21:57:45.262488+0000","last_undegraded":"2023-12-16T21:57:45.262488+0000","last_fullsized":"2023-12-16T21:57:45.262488+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:07:18.558518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.869858+0000","last_change":"2023-12-16T21:57:42.504338+0000","last_active":"2023-12-16T21:57:44.869858+0000","last_peered":"2023-12-16T21:57:44.869858+0000","last_clean":"2023-12-16T21:57:44.869858+0000","last_became_active":"2023-12-16T21:57:42.503125+0000","last_became_peered":"2023-12-16T21:57:42.503125+0000","last_unstale":"2023-12-16T21:57:44.869858+0000","last_undegraded":"2023-12-16T21:57:44.869858+0000","last_fullsized":"2023-12-16T21:57:44.869858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:23:06.394772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262761+0000","last_change":"2023-12-16T21:57:43.499585+0000","last_active":"2023-12-16T21:57:45.262761+0000","last_peered":"2023-12-16T21:57:45.262761+0000","last_clean":"2023-12-16T21:57:45.262761+0000","last_became_active":"2023-12-16T21:57:43.498790+0000","last_became_peered":"2023-12-16T21:57:43.498790+0000","last_unstale":"2023-12-16T21:57:45.262761+0000","last_undegraded":"2023-12-16T21:57:45.262761+0000","last_fullsized":"2023-12-16T21:57:45.262761+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:10.353367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264439+0000","last_change":"2023-12-16T21:57:42.522249+0000","last_active":"2023-12-16T21:57:45.264439+0000","last_peered":"2023-12-16T21:57:45.264439+0000","last_clean":"2023-12-16T21:57:45.264439+0000","last_became_active":"2023-12-16T21:57:42.521585+0000","last_became_peered":"2023-12-16T21:57:42.521585+0000","last_unstale":"2023-12-16T21:57:45.264439+0000","last_undegraded":"2023-12-16T21:57:45.264439+0000","last_fullsized":"2023-12-16T21:57:45.264439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:22.825415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:44.457737+0000","last_change":"2023-12-16T21:57:43.495929+0000","last_active":"2023-12-16T21:57:44.457737+0000","last_peered":"2023-12-16T21:57:44.457737+0000","last_clean":"2023-12-16T21:57:44.457737+0000","last_became_active":"2023-12-16T21:57:43.494696+0000","last_became_peered":"2023-12-16T21:57:43.494696+0000","last_unstale":"2023-12-16T21:57:44.457737+0000","last_undegraded":"2023-12-16T21:57:44.457737+0000","last_fullsized":"2023-12-16T21:57:44.457737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:49:23.836588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262280+0000","last_change":"2023-12-16T21:57:43.496545+0000","last_active":"2023-12-16T21:57:45.262280+0000","last_peered":"2023-12-16T21:57:45.262280+0000","last_clean":"2023-12-16T21:57:45.262280+0000","last_became_active":"2023-12-16T21:57:43.495846+0000","last_became_peered":"2023-12-16T21:57:43.495846+0000","last_unstale":"2023-12-16T21:57:45.262280+0000","last_undegraded":"2023-12-16T21:57:45.262280+0000","last_fullsized":"2023-12-16T21:57:45.262280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264212+0000","last_change":"2023-12-16T21:57:42.524077+0000","last_active":"2023-12-16T21:57:45.264212+0000","last_peered":"2023-12-16T21:57:45.264212+0000","last_clean":"2023-12-16T21:57:45.264212+0000","last_became_active":"2023-12-16T21:57:42.523590+0000","last_became_peered":"2023-12-16T21:57:42.523590+0000","last_unstale":"2023-12-16T21:57:45.264212+0000","last_undegraded":"2023-12-16T21:57:45.264212+0000","last_fullsized":"2023-12-16T21:57:45.264212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:35:57.160385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.262278+0000","last_change":"2023-12-16T21:57:43.508947+0000","last_active":"2023-12-16T21:57:45.262278+0000","last_peered":"2023-12-16T21:57:45.262278+0000","last_clean":"2023-12-16T21:57:45.262278+0000","last_became_active":"2023-12-16T21:57:43.508424+0000","last_became_peered":"2023-12-16T21:57:43.508424+0000","last_unstale":"2023-12-16T21:57:45.262278+0000","last_undegraded":"2023-12-16T21:57:45.262278+0000","last_fullsized":"2023-12-16T21:57:45.262278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:11:36.860587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.264760+0000","last_change":"2023-12-16T21:57:42.522477+0000","last_active":"2023-12-16T21:57:45.264760+0000","last_peered":"2023-12-16T21:57:45.264760+0000","last_clean":"2023-12-16T21:57:45.264760+0000","last_became_active":"2023-12-16T21:57:42.521732+0000","last_became_peered":"2023-12-16T21:57:42.521732+0000","last_unstale":"2023-12-16T21:57:45.264760+0000","last_undegraded":"2023-12-16T21:57:45.264760+0000","last_fullsized":"2023-12-16T21:57:45.264760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:50:07.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.252926+0000","last_change":"2023-12-16T21:57:43.481470+0000","last_active":"2023-12-16T21:57:45.252926+0000","last_peered":"2023-12-16T21:57:45.252926+0000","last_clean":"2023-12-16T21:57:45.252926+0000","last_became_active":"2023-12-16T21:57:43.480541+0000","last_became_peered":"2023-12-16T21:57:43.480541+0000","last_unstale":"2023-12-16T21:57:45.252926+0000","last_undegraded":"2023-12-16T21:57:45.252926+0000","last_fullsized":"2023-12-16T21:57:45.252926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:43:24.033743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263338+0000","last_change":"2023-12-16T21:57:42.512776+0000","last_active":"2023-12-16T21:57:45.263338+0000","last_peered":"2023-12-16T21:57:45.263338+0000","last_clean":"2023-12-16T21:57:45.263338+0000","last_became_active":"2023-12-16T21:57:42.512393+0000","last_became_peered":"2023-12-16T21:57:42.512393+0000","last_unstale":"2023-12-16T21:57:45.263338+0000","last_undegraded":"2023-12-16T21:57:45.263338+0000","last_fullsized":"2023-12-16T21:57:45.263338+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.263375+0000","last_change":"2023-12-16T21:57:43.499571+0000","last_active":"2023-12-16T21:57:45.263375+0000","last_peered":"2023-12-16T21:57:45.263375+0000","last_clean":"2023-12-16T21:57:45.263375+0000","last_became_active":"2023-12-16T21:57:43.498880+0000","last_became_peered":"2023-12-16T21:57:43.498880+0000","last_unstale":"2023-12-16T21:57:45.263375+0000","last_undegraded":"2023-12-16T21:57:45.263375+0000","last_fullsized":"2023-12-16T21:57:45.263375+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:42.463215+0000","last_clean_scrub_stamp":"2023-12-16T21:57:42.463215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:13.579012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-12-16T21:57:45.253109+0000","last_change":"2023-12-16T21:57:42.522619+0000","last_active":"2023-12-16T21:57:45.253109+0000","last_peered":"2023-12-16T21:57:45.253109+0000","last_clean":"2023-12-16T21:57:45.253109+0000","last_became_active":"2023-12-16T21:57:42.522259+0000","last_became_peered":"2023-12-16T21:57:42.522259+0000","last_unstale":"2023-12-16T21:57:45.253109+0000","last_undegraded":"2023-12-16T21:57:45.253109+0000","last_fullsized":"2023-12-16T21:57:45.253109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:41.452042+0000","last_clean_scrub_stamp":"2023-12-16T21:57:41.452042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:29:04.051005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476743,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":827392,"data_stored":670753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]}]},{"osd":6,"up_from":16,"seq":68719476742,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]}]},{"osd":5,"up_from":16,"seq":68719476743,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":79893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]}]},{"osd":4,"up_from":16,"seq":68719476741,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":79929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":241664,"data_stored":81415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":237568,"data_stored":79963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.363}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2869999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":670809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":79803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.212}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:57:50.557 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-16T21:57:50.557 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-16T21:57:50.557 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-16T21:57:50.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-16T21:57:50.717 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe87a61c640 1 Processor -- start 2023-12-16T21:57:50.717 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe87a61c640 1 -- start start 2023-12-16T21:57:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe87a61c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8740b35b0 0x7fe8740d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe87a61c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740b3c70 0x7fe87415f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe87a61c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740d3880 con 0x7fe8740cf780 2023-12-16T21:57:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe87a61c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8740d4090 con 0x7fe8740b35b0 2023-12-16T21:57:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe87a61c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740cdc80 con 0x7fe8740b3c70 2023-12-16T21:57:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe878bb9640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe8740cf780 0x7fe8740ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51258/0 (socket says 172.21.15.28:51258) 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe878bb9640 1 -- 172.21.15.28:0/1444936145 learned_addr learned my addr 172.21.15.28:0/1444936145 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.716+0000 7fe873fff640 1 --2- 172.21.15.28:0/1444936145 >> v2:172.21.15.28:3300/0 conn(0x7fe8740b35b0 0x7fe8740d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe872ffd640 1 -- 172.21.15.28:0/1444936145 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380121981 0 0) 0x7fe8740d3880 con 0x7fe8740cf780 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe8737fe640 1 --2- 172.21.15.28:0/1444936145 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740b3c70 0x7fe87415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe872ffd640 1 -- 172.21.15.28:0/1444936145 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe85c003530 con 0x7fe8740cf780 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe873fff640 1 -- 172.21.15.28:0/1444936145 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740b3c70 msgr2=0x7fe87415f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe873fff640 1 --2- 172.21.15.28:0/1444936145 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740b3c70 0x7fe87415f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe873fff640 1 -- 172.21.15.28:0/1444936145 >> v1:172.21.15.28:6789/0 conn(0x7fe8740cf780 legacy=0x7fe8740ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe873fff640 1 -- 172.21.15.28:0/1444936145 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8740d3ad0 con 0x7fe8740b35b0 2023-12-16T21:57:50.721 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe873fff640 1 --2- 172.21.15.28:0/1444936145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe8740d0f50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe8640097c0 tx=0x7fe86401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477508a68b430386 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.721 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe872ffd640 1 -- 172.21.15.28:0/1444936145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8640ab680 con 0x7fe8740b35b0 2023-12-16T21:57:50.721 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe872ffd640 1 -- 172.21.15.28:0/1444936145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8640ab7e0 con 0x7fe8740b35b0 2023-12-16T21:57:50.721 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe872ffd640 1 -- 172.21.15.28:0/1444936145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8640abb50 con 0x7fe8740b35b0 2023-12-16T21:57:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe87a61c640 1 -- 172.21.15.28:0/1444936145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 msgr2=0x7fe8740d0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe87a61c640 1 --2- 172.21.15.28:0/1444936145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe8740d0f50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe8640097c0 tx=0x7fe86401c480 comp rx=0 tx=0).stop 2023-12-16T21:57:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe87a61c640 1 -- 172.21.15.28:0/1444936145 shutdown_connections 2023-12-16T21:57:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe87a61c640 1 --2- 172.21.15.28:0/1444936145 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740b3c70 0x7fe87415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe87a61c640 1 --2- 172.21.15.28:0/1444936145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe8740d0f50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe87a61c640 1 -- 172.21.15.28:0/1444936145 >> 172.21.15.28:0/1444936145 conn(0x7fe87409c4b0 msgr2=0x7fe8740d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:50.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe87a61c640 1 -- 172.21.15.28:0/1444936145 shutdown_connections 2023-12-16T21:57:50.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.720+0000 7fe87a61c640 1 -- 172.21.15.28:0/1444936145 wait complete. 2023-12-16T21:57:50.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 Processor -- start 2023-12-16T21:57:50.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 -- start start 2023-12-16T21:57:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe87415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8740b3c70 0x7fe87409d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740cf780 0x7fe87409d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740b62a0 con 0x7fe8740b35b0 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe87415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740a05f0 con 0x7fe8740b3c70 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87409fd50 con 0x7fe8740cf780 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe87415b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46202/0 (socket says 172.21.15.28:46202) 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 -- 172.21.15.28:0/3470221140 learned_addr learned my addr 172.21.15.28:0/3470221140 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe878bb9640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740cf780 0x7fe87409d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 -- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740cf780 msgr2=0x7fe87409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740cf780 0x7fe87409d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 -- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8740b3c70 msgr2=0x7fe87409d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8740b3c70 0x7fe87409d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 -- 172.21.15.28:0/3470221140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe864009420 con 0x7fe8740b35b0 2023-12-16T21:57:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe873fff640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe87415b930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe864000c10 tx=0x7fe864019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe870ff9640 1 -- 172.21.15.28:0/3470221140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864004510 con 0x7fe8740b35b0 2023-12-16T21:57:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe870ff9640 1 -- 172.21.15.28:0/3470221140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8640ab950 con 0x7fe8740b35b0 2023-12-16T21:57:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8740d0120 con 0x7fe8740b35b0 2023-12-16T21:57:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe870ff9640 1 -- 172.21.15.28:0/3470221140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8640abcc0 con 0x7fe8740b35b0 2023-12-16T21:57:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.724+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8740a0310 con 0x7fe8740b35b0 2023-12-16T21:57:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.728+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe874069440 con 0x7fe8740b35b0 2023-12-16T21:57:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.728+0000 7fe870ff9640 1 -- 172.21.15.28:0/3470221140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe864003620 con 0x7fe8740b35b0 2023-12-16T21:57:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.728+0000 7fe870ff9640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe860066470 0x7fe860068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.728+0000 7fe870ff9640 1 -- 172.21.15.28:0/3470221140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe86411e250 con 0x7fe8740b35b0 2023-12-16T21:57:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.728+0000 7fe8737fe640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe860066470 0x7fe860068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.728+0000 7fe8737fe640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe860066470 0x7fe860068930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe8740a10b0 tx=0x7fe868008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.732+0000 7fe870ff9640 1 -- 172.21.15.28:0/3470221140 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe8640eb630 con 0x7fe8740b35b0 2023-12-16T21:57:50.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.912+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe8740d0720 con 0x7fe8740b35b0 2023-12-16T21:57:50.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.912+0000 7fe870ff9640 1 -- 172.21.15.28:0/3470221140 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe8640eec50 con 0x7fe8740b35b0 2023-12-16T21:57:50.914 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:50.914 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-16T21:57:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe860066470 msgr2=0x7fe860068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe860066470 0x7fe860068930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe8740a10b0 tx=0x7fe868008040 comp rx=0 tx=0).stop 2023-12-16T21:57:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 msgr2=0x7fe87415b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe87415b930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe864000c10 tx=0x7fe864019220 comp rx=0 tx=0).stop 2023-12-16T21:57:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 shutdown_connections 2023-12-16T21:57:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe860066470 0x7fe860068930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8740cf780 0x7fe87409d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8740b3c70 0x7fe87409d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 --2- 172.21.15.28:0/3470221140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740b35b0 0x7fe87415b930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:50.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 >> 172.21.15.28:0/3470221140 conn(0x7fe87409c4b0 msgr2=0x7fe8740b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:50.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 shutdown_connections 2023-12-16T21:57:50.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:50.916+0000 7fe87a61c640 1 -- 172.21.15.28:0/3470221140 wait complete. 2023-12-16T21:57:50.931 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-16T21:57:50.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2c933640 1 Processor -- start 2023-12-16T21:57:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2c933640 1 -- start start 2023-12-16T21:57:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2c933640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6e280b35b0 0x7f6e280d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2c933640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280b3c70 0x7f6e2815f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2c933640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e280cf780 0x7f6e280cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2c933640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e280d4090 con 0x7f6e280cf780 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2c933640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e280d0130 con 0x7f6e280b35b0 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2c933640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280d3880 con 0x7f6e280b3c70 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e280cf780 0x7f6e280cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e280cf780 0x7f6e280cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46216/0 (socket says 172.21.15.28:46216) 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 -- 172.21.15.28:0/3252996176 learned_addr learned my addr 172.21.15.28:0/3252996176 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 -- 172.21.15.28:0/3252996176 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280b3c70 msgr2=0x7f6e2815f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e25d9b640 1 --2- 172.21.15.28:0/3252996176 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280b3c70 0x7f6e2815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2659c640 1 --2- 172.21.15.28:0/3252996176 >> v2:172.21.15.122:3300/0 conn(0x7f6e280b35b0 0x7f6e280d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 --2- 172.21.15.28:0/3252996176 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280b3c70 0x7f6e2815f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 -- 172.21.15.28:0/3252996176 >> v2:172.21.15.122:3300/0 conn(0x7f6e280b35b0 msgr2=0x7f6e280d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 --2- 172.21.15.28:0/3252996176 >> v2:172.21.15.122:3300/0 conn(0x7f6e280b35b0 0x7f6e280d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 -- 172.21.15.28:0/3252996176 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e280cdc80 con 0x7f6e280cf780 2023-12-16T21:57:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2659c640 1 --2- 172.21.15.28:0/3252996176 >> v2:172.21.15.122:3300/0 conn(0x7f6e280b35b0 0x7f6e280d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e26d9d640 1 --2- 172.21.15.28:0/3252996176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280cf780 0x7f6e280cfbf0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6e2808c9e0 tx=0x7f6e2001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d248ce8e0adbef6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2559a640 1 -- 172.21.15.28:0/3252996176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e200a5850 con 0x7f6e280cf780 2023-12-16T21:57:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2559a640 1 -- 172.21.15.28:0/3252996176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e200a59b0 con 0x7f6e280cf780 2023-12-16T21:57:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.055+0000 7f6e2559a640 1 -- 172.21.15.28:0/3252996176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e200a5d40 con 0x7f6e280cf780 2023-12-16T21:57:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- 172.21.15.28:0/3252996176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280cf780 msgr2=0x7f6e280cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3252996176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280cf780 0x7f6e280cfbf0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6e2808c9e0 tx=0x7f6e2001c9d0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- 172.21.15.28:0/3252996176 shutdown_connections 2023-12-16T21:57:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3252996176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280cf780 0x7f6e280cfbf0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3252996176 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280b3c70 0x7f6e2815f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3252996176 >> v2:172.21.15.122:3300/0 conn(0x7f6e280b35b0 0x7f6e280d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- 172.21.15.28:0/3252996176 >> 172.21.15.28:0/3252996176 conn(0x7f6e2809c4b0 msgr2=0x7f6e280d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- 172.21.15.28:0/3252996176 shutdown_connections 2023-12-16T21:57:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- 172.21.15.28:0/3252996176 wait complete. 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 Processor -- start 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- start start 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280b35b0 0x7f6e280c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e280b3c70 0x7f6e2815a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280cf780 0x7f6e2815a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280cd190 con 0x7f6e280b35b0 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280cc610 con 0x7f6e280b3c70 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e280cc8f0 con 0x7f6e280cf780 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280b35b0 0x7f6e280c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280b35b0 0x7f6e280c8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46224/0 (socket says 172.21.15.28:46224) 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 -- 172.21.15.28:0/3004061300 learned_addr learned my addr 172.21.15.28:0/3004061300 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 -- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280cf780 msgr2=0x7f6e2815a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280cf780 0x7f6e2815a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 -- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e280b3c70 msgr2=0x7f6e2815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e280b3c70 0x7f6e2815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 -- 172.21.15.28:0/3004061300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e20009b30 con 0x7f6e280b35b0 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2659c640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280b35b0 0x7f6e280c8f00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800c7c0 tx=0x7f6e1800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e16ffd640 1 -- 172.21.15.28:0/3004061300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18007a30 con 0x7f6e280b35b0 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e16ffd640 1 -- 172.21.15.28:0/3004061300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e18007b90 con 0x7f6e280b35b0 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e16ffd640 1 -- 172.21.15.28:0/3004061300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1800ce70 con 0x7f6e280b35b0 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e280ccbd0 con 0x7f6e280b35b0 2023-12-16T21:57:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2815b350 con 0x7f6e280b35b0 2023-12-16T21:57:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.059+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e28069440 con 0x7f6e280b35b0 2023-12-16T21:57:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.063+0000 7f6e16ffd640 1 -- 172.21.15.28:0/3004061300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6e180113f0 con 0x7f6e280b35b0 2023-12-16T21:57:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.063+0000 7f6e16ffd640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6e08066470 0x7f6e08068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.063+0000 7f6e16ffd640 1 -- 172.21.15.28:0/3004061300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e18084070 con 0x7f6e280b35b0 2023-12-16T21:57:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.063+0000 7f6e25d9b640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6e08066470 0x7f6e08068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.063+0000 7f6e25d9b640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6e08066470 0x7f6e08068930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6e280cd790 tx=0x7f6e1c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.067+0000 7f6e16ffd640 1 -- 172.21.15.28:0/3004061300 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6e18051400 con 0x7f6e280b35b0 2023-12-16T21:57:51.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.223+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e280d0720 con 0x7f6e280b35b0 2023-12-16T21:57:51.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.223+0000 7f6e16ffd640 1 -- 172.21.15.28:0/3004061300 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f6e1801f090 con 0x7f6e280b35b0 2023-12-16T21:57:51.225 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:51.226 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:51.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6e08066470 msgr2=0x7f6e08068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6e08066470 0x7f6e08068930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6e280cd790 tx=0x7f6e1c008040 comp rx=0 tx=0).stop 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280b35b0 msgr2=0x7f6e280c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280b35b0 0x7f6e280c8f00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e1800c7c0 tx=0x7f6e1800cc80 comp rx=0 tx=0).stop 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 shutdown_connections 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6e08066470 0x7f6e08068930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6e280cf780 0x7f6e2815a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6e280b3c70 0x7f6e2815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 --2- 172.21.15.28:0/3004061300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e280b35b0 0x7f6e280c8f00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 >> 172.21.15.28:0/3004061300 conn(0x7f6e2809c4b0 msgr2=0x7f6e280b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 shutdown_connections 2023-12-16T21:57:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.227+0000 7f6e2c933640 1 -- 172.21.15.28:0/3004061300 wait complete. 2023-12-16T21:57:51.244 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:57:51.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:57:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62ae9bb640 1 Processor -- start 2023-12-16T21:57:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62ae9bb640 1 -- start start 2023-12-16T21:57:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62ae9bb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62a81449d0 0x7f62a806a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62ae9bb640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f62a815f5b0 0x7f62a80cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62ae9bb640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a80cd630 0x7f62a8071590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62ae9bb640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a80cc6a0 con 0x7f62a81449d0 2023-12-16T21:57:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62ae9bb640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a8144db0 con 0x7f62a815f5b0 2023-12-16T21:57:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62ae9bb640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a806ac30 con 0x7f62a80cd630 2023-12-16T21:57:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a7fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62a81449d0 0x7f62a806a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a7fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62a81449d0 0x7f62a806a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46232/0 (socket says 172.21.15.28:46232) 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a7fff640 1 -- 172.21.15.28:0/694461136 learned_addr learned my addr 172.21.15.28:0/694461136 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a77fe640 1 --2- 172.21.15.28:0/694461136 >> v2:172.21.15.122:3300/0 conn(0x7f62a815f5b0 0x7f62a80cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a7fff640 1 -- 172.21.15.28:0/694461136 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a80cd630 msgr2=0x7f62a8071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62acf58640 1 --2- 172.21.15.28:0/694461136 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a80cd630 0x7f62a8071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a7fff640 1 --2- 172.21.15.28:0/694461136 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a80cd630 0x7f62a8071590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a7fff640 1 -- 172.21.15.28:0/694461136 >> v2:172.21.15.122:3300/0 conn(0x7f62a815f5b0 msgr2=0x7f62a80cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a7fff640 1 --2- 172.21.15.28:0/694461136 >> v2:172.21.15.122:3300/0 conn(0x7f62a815f5b0 0x7f62a80cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a7fff640 1 -- 172.21.15.28:0/694461136 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a806af10 con 0x7f62a81449d0 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62a77fe640 1 --2- 172.21.15.28:0/694461136 >> v2:172.21.15.122:3300/0 conn(0x7f62a815f5b0 0x7f62a80cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.359+0000 7f62acf58640 1 --2- 172.21.15.28:0/694461136 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a80cd630 0x7f62a8071590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:51.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62a7fff640 1 --2- 172.21.15.28:0/694461136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a81449d0 0x7f62a806a530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f62a80622f0 tx=0x7f629801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8de534194e14fd7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62a6ffd640 1 -- 172.21.15.28:0/694461136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62980a5760 con 0x7f62a81449d0 2023-12-16T21:57:51.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62a6ffd640 1 -- 172.21.15.28:0/694461136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62980a58c0 con 0x7f62a81449d0 2023-12-16T21:57:51.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62a6ffd640 1 -- 172.21.15.28:0/694461136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62980a5c50 con 0x7f62a81449d0 2023-12-16T21:57:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 -- 172.21.15.28:0/694461136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a81449d0 msgr2=0x7f62a806a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/694461136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a81449d0 0x7f62a806a530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f62a80622f0 tx=0x7f629801c9d0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 -- 172.21.15.28:0/694461136 shutdown_connections 2023-12-16T21:57:51.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/694461136 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a80cd630 0x7f62a8071590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/694461136 >> v2:172.21.15.122:3300/0 conn(0x7f62a815f5b0 0x7f62a80cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/694461136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a81449d0 0x7f62a806a530 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 -- 172.21.15.28:0/694461136 >> 172.21.15.28:0/694461136 conn(0x7f62a806d510 msgr2=0x7f62a806d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:51.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 -- 172.21.15.28:0/694461136 shutdown_connections 2023-12-16T21:57:51.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 -- 172.21.15.28:0/694461136 wait complete. 2023-12-16T21:57:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 Processor -- start 2023-12-16T21:57:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.363+0000 7f62ae9bb640 1 -- start start 2023-12-16T21:57:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a80cd630 0x7f62a813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a81449d0 0x7f62a8157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a815f5b0 0x7f62a8164620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a8158ec0 con 0x7f62a815f5b0 2023-12-16T21:57:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a813fe40 con 0x7f62a80cd630 2023-12-16T21:57:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a813f000 con 0x7f62a81449d0 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a815f5b0 0x7f62a8164620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a815f5b0 0x7f62a8164620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46238/0 (socket says 172.21.15.28:46238) 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 -- 172.21.15.28:0/279111752 learned_addr learned my addr 172.21.15.28:0/279111752 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a7fff640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a80cd630 0x7f62a813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a77fe640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a81449d0 0x7f62a8157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 -- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a81449d0 msgr2=0x7f62a8157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a81449d0 0x7f62a8157dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 -- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a80cd630 msgr2=0x7f62a813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a80cd630 0x7f62a813eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 -- 172.21.15.28:0/279111752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6298009b30 con 0x7f62a815f5b0 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62acf58640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a815f5b0 0x7f62a8164620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c7f0 tx=0x7f62a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a4ff9640 1 -- 172.21.15.28:0/279111752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0007820 con 0x7f62a815f5b0 2023-12-16T21:57:51.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a4ff9640 1 -- 172.21.15.28:0/279111752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a00079c0 con 0x7f62a815f5b0 2023-12-16T21:57:51.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a813f230 con 0x7f62a815f5b0 2023-12-16T21:57:51.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a4ff9640 1 -- 172.21.15.28:0/279111752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0007d70 con 0x7f62a815f5b0 2023-12-16T21:57:51.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a813f7f0 con 0x7f62a815f5b0 2023-12-16T21:57:51.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a80667c0 con 0x7f62a815f5b0 2023-12-16T21:57:51.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a4ff9640 1 -- 172.21.15.28:0/279111752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f62a00113f0 con 0x7f62a815f5b0 2023-12-16T21:57:51.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a4ff9640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62880664a0 0x7f6288068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a4ff9640 1 -- 172.21.15.28:0/279111752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62a00841b0 con 0x7f62a815f5b0 2023-12-16T21:57:51.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a7fff640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62880664a0 0x7f6288068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.367+0000 7f62a7fff640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62880664a0 0x7f6288068960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f62980ae000 tx=0x7f62980a7070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.371+0000 7f62a4ff9640 1 -- 172.21.15.28:0/279111752 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f62a0051630 con 0x7f62a815f5b0 2023-12-16T21:57:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.523+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f62a80674d0 con 0x7f62a815f5b0 2023-12-16T21:57:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62a4ff9640 1 -- 172.21.15.28:0/279111752 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62a0054c50 con 0x7f62a815f5b0 2023-12-16T21:57:51.528 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:57:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62880664a0 msgr2=0x7f6288068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62880664a0 0x7f6288068960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f62980ae000 tx=0x7f62980a7070 comp rx=0 tx=0).stop 2023-12-16T21:57:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a815f5b0 msgr2=0x7f62a8164620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a815f5b0 0x7f62a8164620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c7f0 tx=0x7f62a000ccb0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 shutdown_connections 2023-12-16T21:57:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62880664a0 0x7f6288068960 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a815f5b0 0x7f62a8164620 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62a81449d0 0x7f62a8157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 --2- 172.21.15.28:0/279111752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62a80cd630 0x7f62a813eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 >> 172.21.15.28:0/279111752 conn(0x7f62a806d510 msgr2=0x7f62a8140ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 shutdown_connections 2023-12-16T21:57:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.527+0000 7f62ae9bb640 1 -- 172.21.15.28:0/279111752 wait complete. 2023-12-16T21:57:51.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:57:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fed535640 1 Processor -- start 2023-12-16T21:57:51.717 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fed535640 1 -- start start 2023-12-16T21:57:51.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fed535640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7fe80cba10 0x7f7fe815be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fed535640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7fe80b42d0 0x7f7fe80b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fed535640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe80b4c60 con 0x7f7fe8158ca0 2023-12-16T21:57:51.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fed535640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe815b670 con 0x7f7fe80cba10 2023-12-16T21:57:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fed535640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe815f410 con 0x7f7fe80b42d0 2023-12-16T21:57:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fe67fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7fe80cba10 0x7f7fe815be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fe67fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7fe80cba10 0x7f7fe815be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46248/0 (socket says 172.21.15.28:46248) 2023-12-16T21:57:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fe67fc640 1 -- 172.21.15.28:0/894039934 learned_addr learned my addr 172.21.15.28:0/894039934 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fe77fe640 1 --2- 172.21.15.28:0/894039934 >> v2:172.21.15.122:3300/0 conn(0x7f7fe80b42d0 0x7f7fe80b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fe67fc640 1 -- 172.21.15.28:0/894039934 >> v2:172.21.15.122:3300/0 conn(0x7f7fe80b42d0 msgr2=0x7f7fe80b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fe67fc640 1 --2- 172.21.15.28:0/894039934 >> v2:172.21.15.122:3300/0 conn(0x7f7fe80b42d0 0x7f7fe80b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fe67fc640 1 -- 172.21.15.28:0/894039934 >> v1:172.21.15.122:6789/0 conn(0x7f7fe8158ca0 legacy=0x7f7fe815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.663+0000 7f7fe67fc640 1 -- 172.21.15.28:0/894039934 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe815f6f0 con 0x7f7fe80cba10 2023-12-16T21:57:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe67fc640 1 --2- 172.21.15.28:0/894039934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80cba10 0x7f7fe815be00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7fe80cb970 tx=0x7f7fdc01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee7215b54e4378c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe5ffb640 1 -- 172.21.15.28:0/894039934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0ac630 con 0x7f7fe80cba10 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe5ffb640 1 -- 172.21.15.28:0/894039934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc0ac790 con 0x7f7fe80cba10 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe5ffb640 1 -- 172.21.15.28:0/894039934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0acb20 con 0x7f7fe80cba10 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- 172.21.15.28:0/894039934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80cba10 msgr2=0x7f7fe815be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 --2- 172.21.15.28:0/894039934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80cba10 0x7f7fe815be00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7fe80cb970 tx=0x7f7fdc01c4c0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- 172.21.15.28:0/894039934 shutdown_connections 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 --2- 172.21.15.28:0/894039934 >> v2:172.21.15.122:3300/0 conn(0x7f7fe80b42d0 0x7f7fe80b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 --2- 172.21.15.28:0/894039934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80cba10 0x7f7fe815be00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- 172.21.15.28:0/894039934 >> 172.21.15.28:0/894039934 conn(0x7f7fe80b0cc0 msgr2=0x7f7fe80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- 172.21.15.28:0/894039934 shutdown_connections 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- 172.21.15.28:0/894039934 wait complete. 2023-12-16T21:57:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 Processor -- start 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- start start 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80b42d0 0x7f7fe8152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fe80cba10 0x7f7fe815dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7fe8158ca0 0x7f7fe813a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe814f750 con 0x7f7fe80b42d0 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe815cc50 con 0x7f7fe80cba10 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe815cf30 con 0x7f7fe8158ca0 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80b42d0 0x7f7fe8152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80b42d0 0x7f7fe8152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46260/0 (socket says 172.21.15.28:46260) 2023-12-16T21:57:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 -- 172.21.15.28:0/1207802914 learned_addr learned my addr 172.21.15.28:0/1207802914 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 -- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7fe8158ca0 msgr2=0x7f7fe813a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7fe8158ca0 0x7f7fe813a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 -- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fe80cba10 msgr2=0x7f7fe815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fe80cba10 0x7f7fe815dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 -- 172.21.15.28:0/1207802914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc0094d0 con 0x7f7fe80b42d0 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fe6ffd640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80b42d0 0x7f7fe8152a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800b410 tx=0x7f7fd800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fcf7fe640 1 -- 172.21.15.28:0/1207802914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8004020 con 0x7f7fe80b42d0 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fcf7fe640 1 -- 172.21.15.28:0/1207802914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd8004180 con 0x7f7fe80b42d0 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe815d210 con 0x7f7fe80b42d0 2023-12-16T21:57:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.667+0000 7f7fcf7fe640 1 -- 172.21.15.28:0/1207802914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8011630 con 0x7f7fe80b42d0 2023-12-16T21:57:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.671+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe815d7d0 con 0x7f7fe80b42d0 2023-12-16T21:57:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.671+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe815d9f0 con 0x7f7fe80b42d0 2023-12-16T21:57:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.671+0000 7f7fcf7fe640 1 -- 172.21.15.28:0/1207802914 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7fd8011790 con 0x7f7fe80b42d0 2023-12-16T21:57:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.671+0000 7f7fcf7fe640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7fd4066470 0x7f7fd4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.671+0000 7f7fcf7fe640 1 -- 172.21.15.28:0/1207802914 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fd8083bc0 con 0x7f7fe80b42d0 2023-12-16T21:57:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.671+0000 7f7fe67fc640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7fd4066470 0x7f7fd4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.671+0000 7f7fe67fc640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7fd4066470 0x7f7fd4068930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0ae000 tx=0x7f7fdc004740 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.675+0000 7f7fcf7fe640 1 -- 172.21.15.28:0/1207802914 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7fd8051040 con 0x7f7fe80b42d0 2023-12-16T21:57:51.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.823+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7fe80693d0 con 0x7f7fe80b42d0 2023-12-16T21:57:51.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.823+0000 7f7fcf7fe640 1 -- 172.21.15.28:0/1207802914 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7fd8015020 con 0x7f7fe80b42d0 2023-12-16T21:57:51.826 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7fd4066470 msgr2=0x7f7fd4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7fd4066470 0x7f7fd4068930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0ae000 tx=0x7f7fdc004740 comp rx=0 tx=0).stop 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80b42d0 msgr2=0x7f7fe8152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80b42d0 0x7f7fe8152a00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800b410 tx=0x7f7fd800b8d0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 shutdown_connections 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7fd4066470 0x7f7fd4068930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7fe8158ca0 0x7f7fe813a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fe80cba10 0x7f7fe815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 --2- 172.21.15.28:0/1207802914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe80b42d0 0x7f7fe8152a00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 >> 172.21.15.28:0/1207802914 conn(0x7f7fe80b0cc0 msgr2=0x7f7fe815aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 shutdown_connections 2023-12-16T21:57:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.827+0000 7f7fed535640 1 -- 172.21.15.28:0/1207802914 wait complete. 2023-12-16T21:57:51.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:57:51.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.955+0000 7f3b46894640 1 Processor -- start 2023-12-16T21:57:51.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b46894640 1 -- start start 2023-12-16T21:57:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b46894640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b4013d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b46894640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b4013f5f0 0x7f3b400ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b46894640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4013d2b0 con 0x7f3b40071d90 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b46894640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b40140520 con 0x7f3b4013f5f0 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b46894640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400cd7f0 con 0x7f3b4015adb0 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b3ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3b40071d90 0x7f3b40072170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51260/0 (socket says 172.21.15.28:51260) 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b44e31640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b4013f5f0 0x7f3b400ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b3ffff640 1 -- 172.21.15.28:0/4289576669 learned_addr learned my addr 172.21.15.28:0/4289576669 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b377fe640 1 --2- 172.21.15.28:0/4289576669 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b4013d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b44e31640 1 -- 172.21.15.28:0/4289576669 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 msgr2=0x7f3b4013d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b44e31640 1 --2- 172.21.15.28:0/4289576669 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b4013d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b44e31640 1 -- 172.21.15.28:0/4289576669 >> v1:172.21.15.28:6789/0 conn(0x7f3b40071d90 legacy=0x7f3b40072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b44e31640 1 -- 172.21.15.28:0/4289576669 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400cdd70 con 0x7f3b4013f5f0 2023-12-16T21:57:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b377fe640 1 --2- 172.21.15.28:0/4289576669 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b4013d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:51.962 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.959+0000 7f3b44e31640 1 --2- 172.21.15.28:0/4289576669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b4013f5f0 0x7f3b400ce200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009d90 tx=0x7f3b3801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665cab3f026c6e74 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3f7fe640 1 -- 172.21.15.28:0/4289576669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b380a59e0 con 0x7f3b4013f5f0 2023-12-16T21:57:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3f7fe640 1 -- 172.21.15.28:0/4289576669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b380a5b40 con 0x7f3b4013f5f0 2023-12-16T21:57:51.963 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3f7fe640 1 -- 172.21.15.28:0/4289576669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b380ad440 con 0x7f3b4013f5f0 2023-12-16T21:57:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- 172.21.15.28:0/4289576669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b4013f5f0 msgr2=0x7f3b400ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 --2- 172.21.15.28:0/4289576669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b4013f5f0 0x7f3b400ce200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009d90 tx=0x7f3b3801c920 comp rx=0 tx=0).stop 2023-12-16T21:57:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- 172.21.15.28:0/4289576669 shutdown_connections 2023-12-16T21:57:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 --2- 172.21.15.28:0/4289576669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b4013f5f0 0x7f3b400ce200 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 --2- 172.21.15.28:0/4289576669 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b4013d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- 172.21.15.28:0/4289576669 >> 172.21.15.28:0/4289576669 conn(0x7f3b400b0cc0 msgr2=0x7f3b400b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- 172.21.15.28:0/4289576669 shutdown_connections 2023-12-16T21:57:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- 172.21.15.28:0/4289576669 wait complete. 2023-12-16T21:57:51.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 Processor -- start 2023-12-16T21:57:51.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- start start 2023-12-16T21:57:51.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b40071d90 0x7f3b4013c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b4013f5f0 0x7f3b4013bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b40154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b40152300 con 0x7f3b40071d90 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b401516c0 con 0x7f3b4013f5f0 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b40139870 con 0x7f3b4015adb0 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b40071d90 0x7f3b4013c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b40071d90 0x7f3b4013c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46276/0 (socket says 172.21.15.28:46276) 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 -- 172.21.15.28:0/3758924067 learned_addr learned my addr 172.21.15.28:0/3758924067 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b377fe640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b4013f5f0 0x7f3b4013bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b44e31640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b40154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 -- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 msgr2=0x7f3b40154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b40154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 -- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b4013f5f0 msgr2=0x7f3b4013bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b4013f5f0 0x7f3b4013bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 -- 172.21.15.28:0/3758924067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b38009a70 con 0x7f3b40071d90 2023-12-16T21:57:51.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b377fe640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b4013f5f0 0x7f3b4013bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3ffff640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b40071d90 0x7f3b4013c9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000c370 tx=0x7f3b3000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3d7fa640 1 -- 172.21.15.28:0/3758924067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30005cf0 con 0x7f3b40071d90 2023-12-16T21:57:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3d7fa640 1 -- 172.21.15.28:0/3758924067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b30005e50 con 0x7f3b40071d90 2023-12-16T21:57:51.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b40151960 con 0x7f3b40071d90 2023-12-16T21:57:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b3d7fa640 1 -- 172.21.15.28:0/3758924067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b300072f0 con 0x7f3b40071d90 2023-12-16T21:57:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.963+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b40151f20 con 0x7f3b40071d90 2023-12-16T21:57:51.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.967+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b400693d0 con 0x7f3b40071d90 2023-12-16T21:57:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.967+0000 7f3b3d7fa640 1 -- 172.21.15.28:0/3758924067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f3b30007450 con 0x7f3b40071d90 2023-12-16T21:57:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.967+0000 7f3b3d7fa640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b20066470 0x7f3b20068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.967+0000 7f3b3d7fa640 1 -- 172.21.15.28:0/3758924067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b30083410 con 0x7f3b40071d90 2023-12-16T21:57:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.967+0000 7f3b377fe640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b20066470 0x7f3b20068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:51.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.967+0000 7f3b377fe640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b20066470 0x7f3b20068930 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3b400b4b20 tx=0x7f3b28008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:51.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:51.971+0000 7f3b3d7fa640 1 -- 172.21.15.28:0/3758924067 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3b30050880 con 0x7f3b40071d90 2023-12-16T21:57:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.127+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3b401520e0 con 0x7f3b40071d90 2023-12-16T21:57:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.127+0000 7f3b3d7fa640 1 -- 172.21.15.28:0/3758924067 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3b30017020 con 0x7f3b40071d90 2023-12-16T21:57:52.130 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:57:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b20066470 msgr2=0x7f3b20068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b20066470 0x7f3b20068930 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3b400b4b20 tx=0x7f3b28008040 comp rx=0 tx=0).stop 2023-12-16T21:57:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b40071d90 msgr2=0x7f3b4013c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b40071d90 0x7f3b4013c9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000c370 tx=0x7f3b3000c830 comp rx=0 tx=0).stop 2023-12-16T21:57:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 shutdown_connections 2023-12-16T21:57:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b20066470 0x7f3b20068930 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b4015adb0 0x7f3b40154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b4013f5f0 0x7f3b4013bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 --2- 172.21.15.28:0/3758924067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b40071d90 0x7f3b4013c9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 >> 172.21.15.28:0/3758924067 conn(0x7f3b400b0cc0 msgr2=0x7f3b4015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 shutdown_connections 2023-12-16T21:57:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.131+0000 7f3b46894640 1 -- 172.21.15.28:0/3758924067 wait complete. 2023-12-16T21:57:52.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:57:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e3d683640 1 Processor -- start 2023-12-16T21:57:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e3d683640 1 -- start start 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e3d683640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e3806eab0 0x7f0e3813d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e3d683640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e3806ee90 0x7f0e3813df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e3d683640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3813e470 con 0x7f0e3815ec80 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e3d683640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3815f9f0 con 0x7f0e3806eab0 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e3d683640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3815b900 con 0x7f0e3806ee90 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e36ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e3806eab0 0x7f0e3813d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e36ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e3806eab0 0x7f0e3813d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46280/0 (socket says 172.21.15.28:46280) 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e377fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0e3815ec80 0x7f0e3815f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51266/0 (socket says 172.21.15.28:51266) 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e36ffd640 1 -- 172.21.15.28:0/1020396830 learned_addr learned my addr 172.21.15.28:0/1020396830 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e2ffff640 1 --2- 172.21.15.28:0/1020396830 >> v2:172.21.15.122:3300/0 conn(0x7f0e3806ee90 0x7f0e3813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e36ffd640 1 -- 172.21.15.28:0/1020396830 >> v2:172.21.15.122:3300/0 conn(0x7f0e3806ee90 msgr2=0x7f0e3813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:52.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/1020396830 >> v2:172.21.15.122:3300/0 conn(0x7f0e3806ee90 0x7f0e3813df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e36ffd640 1 -- 172.21.15.28:0/1020396830 >> v1:172.21.15.28:6789/0 conn(0x7f0e3815ec80 legacy=0x7f0e3815f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e36ffd640 1 -- 172.21.15.28:0/1020396830 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3815bbe0 con 0x7f0e3806eab0 2023-12-16T21:57:52.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.271+0000 7f0e367fc640 1 -- 172.21.15.28:0/1020396830 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717982226 0 0) 0x7f0e3813e470 con 0x7f0e3815ec80 2023-12-16T21:57:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/1020396830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e3813d9f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009870 tx=0x7f0e2801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad528b34571ad9a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e367fc640 1 -- 172.21.15.28:0/1020396830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280ab610 con 0x7f0e3806eab0 2023-12-16T21:57:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e367fc640 1 -- 172.21.15.28:0/1020396830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e280ab770 con 0x7f0e3806eab0 2023-12-16T21:57:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e367fc640 1 -- 172.21.15.28:0/1020396830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280abb00 con 0x7f0e3806eab0 2023-12-16T21:57:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- 172.21.15.28:0/1020396830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 msgr2=0x7f0e3813d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 --2- 172.21.15.28:0/1020396830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e3813d9f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009870 tx=0x7f0e2801c500 comp rx=0 tx=0).stop 2023-12-16T21:57:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- 172.21.15.28:0/1020396830 shutdown_connections 2023-12-16T21:57:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 --2- 172.21.15.28:0/1020396830 >> v2:172.21.15.122:3300/0 conn(0x7f0e3806ee90 0x7f0e3813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 --2- 172.21.15.28:0/1020396830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e3813d9f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- 172.21.15.28:0/1020396830 >> 172.21.15.28:0/1020396830 conn(0x7f0e3806a930 msgr2=0x7f0e3806ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- 172.21.15.28:0/1020396830 shutdown_connections 2023-12-16T21:57:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- 172.21.15.28:0/1020396830 wait complete. 2023-12-16T21:57:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 Processor -- start 2023-12-16T21:57:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- start start 2023-12-16T21:57:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e38139cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e3806ee90 0x7f0e38137460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3815ec80 0x7f0e380b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e381398c0 con 0x7f0e3806eab0 2023-12-16T21:57:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3814d140 con 0x7f0e3815ec80 2023-12-16T21:57:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e3d683640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e380b5dc0 con 0x7f0e3806ee90 2023-12-16T21:57:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e38139cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e38139cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46290/0 (socket says 172.21.15.28:46290) 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 -- 172.21.15.28:0/4172318762 learned_addr learned my addr 172.21.15.28:0/4172318762 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e377fe640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3815ec80 0x7f0e380b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e2ffff640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e3806ee90 0x7f0e38137460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 -- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e3806ee90 msgr2=0x7f0e38137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e3806ee90 0x7f0e38137460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 -- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3815ec80 msgr2=0x7f0e380b53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3815ec80 0x7f0e380b53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 -- 172.21.15.28:0/4172318762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280094d0 con 0x7f0e3806eab0 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e2ffff640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e3806ee90 0x7f0e38137460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.275+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e38139cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e280ad040 tx=0x7f0e2801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e2f7fe640 1 -- 172.21.15.28:0/4172318762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280b4800 con 0x7f0e3806eab0 2023-12-16T21:57:52.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e2f7fe640 1 -- 172.21.15.28:0/4172318762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e280b4980 con 0x7f0e3806eab0 2023-12-16T21:57:52.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e38138f20 con 0x7f0e3806eab0 2023-12-16T21:57:52.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e2f7fe640 1 -- 172.21.15.28:0/4172318762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e280b4d30 con 0x7f0e3806eab0 2023-12-16T21:57:52.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e381394e0 con 0x7f0e3806eab0 2023-12-16T21:57:52.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e38139b30 con 0x7f0e3806eab0 2023-12-16T21:57:52.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e2f7fe640 1 -- 172.21.15.28:0/4172318762 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f0e380d0830 con 0x7f0e3806eab0 2023-12-16T21:57:52.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e2f7fe640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e18066470 0x7f0e18068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e2f7fe640 1 -- 172.21.15.28:0/4172318762 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e2811e690 con 0x7f0e3806eab0 2023-12-16T21:57:52.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e2ffff640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e18066470 0x7f0e18068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.279+0000 7f0e2ffff640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e18066470 0x7f0e18068930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0e380675f0 tx=0x7f0e20008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.283+0000 7f0e2f7fe640 1 -- 172.21.15.28:0/4172318762 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f0e280ebcc0 con 0x7f0e3806eab0 2023-12-16T21:57:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.471+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e38069440 con 0x7f0e3806eab0 2023-12-16T21:57:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.471+0000 7f0e2f7fe640 1 -- 172.21.15.28:0/4172318762 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4932 (secure 0 0 0) 0x7f0e280ab5d0 con 0x7f0e3806eab0 2023-12-16T21:57:52.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:52.474 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:57:42.850798+0000","modified":"2023-12-16T21:57:45.892479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:57:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2023-12-16T21:57:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e18066470 msgr2=0x7f0e18068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e18066470 0x7f0e18068930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0e380675f0 tx=0x7f0e20008040 comp rx=0 tx=0).stop 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 msgr2=0x7f0e38139cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e38139cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0e280ad040 tx=0x7f0e2801ad40 comp rx=0 tx=0).stop 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 shutdown_connections 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e18066470 0x7f0e18068930 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e3815ec80 0x7f0e380b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e3806ee90 0x7f0e38137460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 --2- 172.21.15.28:0/4172318762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3806eab0 0x7f0e38139cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 >> 172.21.15.28:0/4172318762 conn(0x7f0e3806a930 msgr2=0x7f0e3815d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 shutdown_connections 2023-12-16T21:57:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.475+0000 7f0e3d683640 1 -- 172.21.15.28:0/4172318762 wait complete. 2023-12-16T21:57:52.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:52.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.615+0000 7f21e5ed1640 1 Processor -- start 2023-12-16T21:57:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.615+0000 7f21e5ed1640 1 -- start start 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21e5ed1640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00b0d80 0x7f21e0165f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21e5ed1640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e00cc320 con 0x7f21e015b230 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21e5ed1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e00cf9a0 con 0x7f21e00cf510 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21e5ed1640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00b1210 con 0x7f21e00b0d80 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21df7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f21e015b230 0x7f21e00cc7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51282/0 (socket says 172.21.15.28:51282) 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21df7fe640 1 -- 172.21.15.28:0/2036382841 learned_addr learned my addr 172.21.15.28:0/2036382841 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543533011 0 0) 0x7f21e00cc320 con 0x7f21e015b230 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c8003530 con 0x7f21e015b230 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21dffff640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00b0d80 0x7f21e0165f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2096665042 0 0) 0x7f21c8003530 con 0x7f21e015b230 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00b0d80 msgr2=0x7f21e0165f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00b0d80 0x7f21e0165f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 >> v1:172.21.15.122:6789/0 conn(0x7f21e00cf510 legacy=0x7f21e00b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e00cfbf0 con 0x7f21e015b230 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f21d0001680 con 0x7f21e015b230 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 >> v1:172.21.15.28:6789/0 conn(0x7f21e015b230 legacy=0x7f21e00cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c80039b0 0x7f21c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00b0d80 0x7f21c80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c8004d60 0x7f21c80051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00b1210 con 0x7f21c8004d60 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c800bc10 con 0x7f21c80039b0 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c800bdd0 con 0x7f21e00b0d80 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21dffff640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c8004d60 0x7f21c80051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21df7fe640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c80039b0 0x7f21c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21deffd640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00b0d80 0x7f21c80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21dffff640 1 -- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00b0d80 msgr2=0x7f21c80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21dffff640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00b0d80 0x7f21c80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21dffff640 1 -- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c80039b0 msgr2=0x7f21c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21dffff640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c80039b0 0x7f21c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21dffff640 1 -- 172.21.15.28:0/2036382841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c8003530 con 0x7f21c8004d60 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21dffff640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c8004d60 0x7f21c80051e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f21d8007330 tx=0x7f21d801cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70df5a9aebef3dd4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d80a5bb0 con 0x7f21c8004d60 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d80a5d10 con 0x7f21c8004d60 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.619+0000 7f21de7fc640 1 -- 172.21.15.28:0/2036382841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d80ad660 con 0x7f21c8004d60 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2036382841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c8004d60 msgr2=0x7f21c80051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 --2- 172.21.15.28:0/2036382841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c8004d60 0x7f21c80051e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f21d8007330 tx=0x7f21d801cde0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 -- 172.21.15.28:0/2036382841 reap_dead start 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2036382841 shutdown_connections 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2036382841 >> 172.21.15.28:0/2036382841 conn(0x7f21e00d0f30 msgr2=0x7f21e00d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2036382841 shutdown_connections 2023-12-16T21:57:52.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2036382841 wait complete. 2023-12-16T21:57:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 Processor -- start 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- start start 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c8004d60 0x7f21e014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00c9dc0 0x7f21e00c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21e00ca240 0x7f21e00c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00cb420 con 0x7f21e00ca240 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e014e860 con 0x7f21c8004d60 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e00ca810 con 0x7f21e00c9dc0 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21e00ca240 0x7f21e00c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21e00ca240 0x7f21e00c56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46316/0 (socket says 172.21.15.28:46316) 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 -- 172.21.15.28:0/2523360883 learned_addr learned my addr 172.21.15.28:0/2523360883 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21df7fe640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c8004d60 0x7f21e014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21deffd640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00c9dc0 0x7f21e00c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 -- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00c9dc0 msgr2=0x7f21e00c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00c9dc0 0x7f21e00c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 -- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c8004d60 msgr2=0x7f21e014e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c8004d60 0x7f21e014e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 -- 172.21.15.28:0/2523360883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e00caaf0 con 0x7f21e00ca240 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21dffff640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21e00ca240 0x7f21e00c56a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f21d80b0040 tx=0x7f21c8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21c7fff640 1 -- 172.21.15.28:0/2523360883 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d80a59b0 con 0x7f21e00ca240 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21c7fff640 1 -- 172.21.15.28:0/2523360883 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d80a5b10 con 0x7f21e00ca240 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e00cadd0 con 0x7f21e00ca240 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21c7fff640 1 -- 172.21.15.28:0/2523360883 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d80adb40 con 0x7f21e00ca240 2023-12-16T21:57:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.623+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e00c4750 con 0x7f21e00ca240 2023-12-16T21:57:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.627+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e00d0950 con 0x7f21e00ca240 2023-12-16T21:57:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.627+0000 7f21c7fff640 1 -- 172.21.15.28:0/2523360883 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f21d80b4070 con 0x7f21e00ca240 2023-12-16T21:57:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.627+0000 7f21c7fff640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f21cc066470 0x7f21cc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.627+0000 7f21c7fff640 1 -- 172.21.15.28:0/2523360883 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21d811dec0 con 0x7f21e00ca240 2023-12-16T21:57:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.627+0000 7f21df7fe640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f21cc066470 0x7f21cc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.627+0000 7f21df7fe640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f21cc066470 0x7f21cc068930 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21e00cf170 tx=0x7f21d0007040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.631+0000 7f21c7fff640 1 -- 172.21.15.28:0/2523360883 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f21d80eb360 con 0x7f21e00ca240 2023-12-16T21:57:52.790 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.787+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21e00674d0 con 0x7f21e00ca240 2023-12-16T21:57:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21c7fff640 1 -- 172.21.15.28:0/2523360883 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f21d80b2090 con 0x7f21e00ca240 2023-12-16T21:57:52.791 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:52.791 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:52.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f21cc066470 msgr2=0x7f21cc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f21cc066470 0x7f21cc068930 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21e00cf170 tx=0x7f21d0007040 comp rx=0 tx=0).stop 2023-12-16T21:57:52.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21e00ca240 msgr2=0x7f21e00c56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21e00ca240 0x7f21e00c56a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f21d80b0040 tx=0x7f21c8005800 comp rx=0 tx=0).stop 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 shutdown_connections 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f21cc066470 0x7f21cc068930 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21e00ca240 0x7f21e00c56a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21e00c9dc0 0x7f21e00c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 --2- 172.21.15.28:0/2523360883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21c8004d60 0x7f21e014e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 >> 172.21.15.28:0/2523360883 conn(0x7f21e00d0f30 msgr2=0x7f21e006e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 shutdown_connections 2023-12-16T21:57:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.791+0000 7f21e5ed1640 1 -- 172.21.15.28:0/2523360883 wait complete. 2023-12-16T21:57:52.810 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:57:52.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:57:52.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e84074640 1 Processor -- start 2023-12-16T21:57:52.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e84074640 1 -- start start 2023-12-16T21:57:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e84074640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e7c140140 0x7f0e7c0b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e84074640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e7c140520 0x7f0e7c15d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e84074640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c143a50 con 0x7f0e7c06a530 2023-12-16T21:57:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e84074640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7c142ed0 con 0x7f0e7c140520 2023-12-16T21:57:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e84074640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7c1431b0 con 0x7f0e7c140140 2023-12-16T21:57:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e8160f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e7c140520 0x7f0e7c15d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e8160f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e7c140520 0x7f0e7c15d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46332/0 (socket says 172.21.15.28:46332) 2023-12-16T21:57:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e8160f640 1 -- 172.21.15.28:0/4091239167 learned_addr learned my addr 172.21.15.28:0/4091239167 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e81e10640 1 --2- 172.21.15.28:0/4091239167 >> v2:172.21.15.122:3300/0 conn(0x7f0e7c140140 0x7f0e7c0b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e8160f640 1 -- 172.21.15.28:0/4091239167 >> v2:172.21.15.122:3300/0 conn(0x7f0e7c140140 msgr2=0x7f0e7c0b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e8160f640 1 --2- 172.21.15.28:0/4091239167 >> v2:172.21.15.122:3300/0 conn(0x7f0e7c140140 0x7f0e7c0b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e8160f640 1 -- 172.21.15.28:0/4091239167 >> v1:172.21.15.122:6789/0 conn(0x7f0e7c06a530 legacy=0x7f0e7c1611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.935+0000 7f0e8160f640 1 -- 172.21.15.28:0/4091239167 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c143490 con 0x7f0e7c140520 2023-12-16T21:57:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e8160f640 1 --2- 172.21.15.28:0/4091239167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140520 0x7f0e7c15d890 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c06e620 tx=0x7f0e7001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b691b6395672936 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e80e0e640 1 -- 172.21.15.28:0/4091239167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700ab640 con 0x7f0e7c140520 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e80e0e640 1 -- 172.21.15.28:0/4091239167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e700a7070 con 0x7f0e7c140520 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e80e0e640 1 -- 172.21.15.28:0/4091239167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700ab9c0 con 0x7f0e7c140520 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- 172.21.15.28:0/4091239167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140520 msgr2=0x7f0e7c15d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 --2- 172.21.15.28:0/4091239167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140520 0x7f0e7c15d890 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c06e620 tx=0x7f0e7001c440 comp rx=0 tx=0).stop 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- 172.21.15.28:0/4091239167 shutdown_connections 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 --2- 172.21.15.28:0/4091239167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140520 0x7f0e7c15d890 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 --2- 172.21.15.28:0/4091239167 >> v2:172.21.15.122:3300/0 conn(0x7f0e7c140140 0x7f0e7c0b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- 172.21.15.28:0/4091239167 >> 172.21.15.28:0/4091239167 conn(0x7f0e7c0b1db0 msgr2=0x7f0e7c0b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- 172.21.15.28:0/4091239167 shutdown_connections 2023-12-16T21:57:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- 172.21.15.28:0/4091239167 wait complete. 2023-12-16T21:57:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 Processor -- start 2023-12-16T21:57:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- start start 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e7c06a530 0x7f0e7c15b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140140 0x7f0e7c139800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e7c140520 0x7f0e7c141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c15e380 con 0x7f0e7c140140 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c15e660 con 0x7f0e7c06a530 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e84074640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c15e940 con 0x7f0e7c140520 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.939+0000 7f0e8160f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140140 0x7f0e7c139800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e8160f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140140 0x7f0e7c139800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46336/0 (socket says 172.21.15.28:46336) 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e8160f640 1 -- 172.21.15.28:0/3583585638 learned_addr learned my addr 172.21.15.28:0/3583585638 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e81e10640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e7c06a530 0x7f0e7c15b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e82611640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e7c140520 0x7f0e7c141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e8160f640 1 -- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e7c140520 msgr2=0x7f0e7c141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e8160f640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e7c140520 0x7f0e7c141ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e8160f640 1 -- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e7c06a530 msgr2=0x7f0e7c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e8160f640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e7c06a530 0x7f0e7c15b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e8160f640 1 -- 172.21.15.28:0/3583585638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e700094d0 con 0x7f0e7c140140 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e82611640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e7c140520 0x7f0e7c141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e8160f640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140140 0x7f0e7c139800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0e70019050 tx=0x7f0e7001ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e7a7fc640 1 -- 172.21.15.28:0/3583585638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700be3f0 con 0x7f0e7c140140 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e7a7fc640 1 -- 172.21.15.28:0/3583585638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e700be550 con 0x7f0e7c140140 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7c06aed0 con 0x7f0e7c140140 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e7a7fc640 1 -- 172.21.15.28:0/3583585638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e700be900 con 0x7f0e7c140140 2023-12-16T21:57:52.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7c15ef00 con 0x7f0e7c140140 2023-12-16T21:57:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7c0674d0 con 0x7f0e7c140140 2023-12-16T21:57:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e7a7fc640 1 -- 172.21.15.28:0/3583585638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f0e700c9480 con 0x7f0e7c140140 2023-12-16T21:57:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e7a7fc640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e680664a0 0x7f0e68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e7a7fc640 1 -- 172.21.15.28:0/3583585638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e7011e610 con 0x7f0e7c140140 2023-12-16T21:57:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.943+0000 7f0e81e10640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e680664a0 0x7f0e68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.947+0000 7f0e81e10640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e680664a0 0x7f0e68068960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0e74003fd0 tx=0x7f0e740039a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:52.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:52.947+0000 7f0e7a7fc640 1 -- 172.21.15.28:0/3583585638 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f0e700eba90 con 0x7f0e7c140140 2023-12-16T21:57:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e7c06eb20 con 0x7f0e7c140140 2023-12-16T21:57:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e7a7fc640 1 -- 172.21.15.28:0/3583585638 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e700ef0b0 con 0x7f0e7c140140 2023-12-16T21:57:53.104 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:57:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e680664a0 msgr2=0x7f0e68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e680664a0 0x7f0e68068960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0e74003fd0 tx=0x7f0e740039a0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140140 msgr2=0x7f0e7c139800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140140 0x7f0e7c139800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0e70019050 tx=0x7f0e7001ac80 comp rx=0 tx=0).stop 2023-12-16T21:57:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 shutdown_connections 2023-12-16T21:57:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e680664a0 0x7f0e68068960 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e7c140520 0x7f0e7c141ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e7c140140 0x7f0e7c139800 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.107 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 --2- 172.21.15.28:0/3583585638 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e7c06a530 0x7f0e7c15b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.107 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 >> 172.21.15.28:0/3583585638 conn(0x7f0e7c0b1db0 msgr2=0x7f0e7c0b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:53.107 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 shutdown_connections 2023-12-16T21:57:53.107 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.103+0000 7f0e84074640 1 -- 172.21.15.28:0/3583585638 wait complete. 2023-12-16T21:57:53.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:57:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.239+0000 7f69811bc640 1 Processor -- start 2023-12-16T21:57:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.239+0000 7f69811bc640 1 -- start start 2023-12-16T21:57:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f697c0b42d0 0x7f697c0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c0b4c60 con 0x7f697c158ca0 2023-12-16T21:57:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697c15b670 con 0x7f697c0cba10 2023-12-16T21:57:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f697c15c510 con 0x7f697c0b42d0 2023-12-16T21:57:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f697ad9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f697c158ca0 0x7f697c15eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51290/0 (socket says 172.21.15.28:51290) 2023-12-16T21:57:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f697ad9d640 1 -- 172.21.15.28:0/2889616395 learned_addr learned my addr 172.21.15.28:0/2889616395 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f697b59e640 1 --2- 172.21.15.28:0/2889616395 >> v2:172.21.15.28:3300/0 conn(0x7f697c0b42d0 0x7f697c0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f697b59e640 1 -- 172.21.15.28:0/2889616395 >> v1:172.21.15.122:6789/0 conn(0x7f697c0cba10 legacy=0x7f697c15be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f697b59e640 1 -- 172.21.15.28:0/2889616395 >> v1:172.21.15.28:6789/0 conn(0x7f697c158ca0 legacy=0x7f697c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f697b59e640 1 -- 172.21.15.28:0/2889616395 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697c15f410 con 0x7f697c0b42d0 2023-12-16T21:57:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f6979d9b640 1 -- 172.21.15.28:0/2889616395 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943036515 0 0) 0x7f696c001120 con 0x7f697c158ca0 2023-12-16T21:57:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f697b59e640 1 --2- 172.21.15.28:0/2889616395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c0b42d0 0x7f697c0b4720 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6974009d90 tx=0x7f697401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7d1f1f01193418 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f6979d9b640 1 -- 172.21.15.28:0/2889616395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69740a5900 con 0x7f697c0b42d0 2023-12-16T21:57:53.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f6979d9b640 1 -- 172.21.15.28:0/2889616395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69740a5a60 con 0x7f697c0b42d0 2023-12-16T21:57:53.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f6979d9b640 1 -- 172.21.15.28:0/2889616395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69740a5df0 con 0x7f697c0b42d0 2023-12-16T21:57:53.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 -- 172.21.15.28:0/2889616395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c0b42d0 msgr2=0x7f697c0b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 --2- 172.21.15.28:0/2889616395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c0b42d0 0x7f697c0b4720 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6974009d90 tx=0x7f697401c950 comp rx=0 tx=0).stop 2023-12-16T21:57:53.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 -- 172.21.15.28:0/2889616395 shutdown_connections 2023-12-16T21:57:53.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 --2- 172.21.15.28:0/2889616395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c0b42d0 0x7f697c0b4720 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 -- 172.21.15.28:0/2889616395 >> 172.21.15.28:0/2889616395 conn(0x7f697c0b0cc0 msgr2=0x7f697c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 -- 172.21.15.28:0/2889616395 shutdown_connections 2023-12-16T21:57:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.243+0000 7f69811bc640 1 -- 172.21.15.28:0/2889616395 wait complete. 2023-12-16T21:57:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 Processor -- start 2023-12-16T21:57:53.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 -- start start 2023-12-16T21:57:53.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f697c0b42d0 0x7f697c14cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f697c0cba10 0x7f697c15cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c158ca0 0x7f697c15d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c14efd0 con 0x7f697c158ca0 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c14e6e0 con 0x7f697c0b42d0 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697c14e980 con 0x7f697c0cba10 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c158ca0 0x7f697c15d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c158ca0 0x7f697c15d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46352/0 (socket says 172.21.15.28:46352) 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 -- 172.21.15.28:0/3929778185 learned_addr learned my addr 172.21.15.28:0/3929778185 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697a59c640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f697c0cba10 0x7f697c15cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697ad9d640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f697c0b42d0 0x7f697c14cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 -- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f697c0cba10 msgr2=0x7f697c15cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f697c0cba10 0x7f697c15cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 -- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f697c0b42d0 msgr2=0x7f697c14cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f697c0b42d0 0x7f697c14cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 -- 172.21.15.28:0/3929778185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6974009a70 con 0x7f697c158ca0 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697a59c640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f697c0cba10 0x7f697c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:53.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697ad9d640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f697c0b42d0 0x7f697c14cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:53.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f697b59e640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c158ca0 0x7f697c15d010 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f697401a580 tx=0x7f69740196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69637fe640 1 -- 172.21.15.28:0/3929778185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69740ad950 con 0x7f697c158ca0 2023-12-16T21:57:53.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69637fe640 1 -- 172.21.15.28:0/3929778185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69740adab0 con 0x7f697c158ca0 2023-12-16T21:57:53.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69637fe640 1 -- 172.21.15.28:0/3929778185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69740ade40 con 0x7f697c158ca0 2023-12-16T21:57:53.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697c14ec60 con 0x7f697c158ca0 2023-12-16T21:57:53.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c152a40 con 0x7f697c158ca0 2023-12-16T21:57:53.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697c072650 con 0x7f697c158ca0 2023-12-16T21:57:53.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69637fe640 1 -- 172.21.15.28:0/3929778185 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f69740b4050 con 0x7f697c158ca0 2023-12-16T21:57:53.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.247+0000 7f69637fe640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6968066470 0x7f6968068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.251+0000 7f69637fe640 1 -- 172.21.15.28:0/3929778185 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f697411f940 con 0x7f697c158ca0 2023-12-16T21:57:53.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.251+0000 7f697ad9d640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6968066470 0x7f6968068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.251+0000 7f697ad9d640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6968066470 0x7f6968068930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f696c000f70 tx=0x7f696c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.254 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.251+0000 7f69637fe640 1 -- 172.21.15.28:0/3929778185 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f69740ecda0 con 0x7f697c158ca0 2023-12-16T21:57:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f697c0693d0 con 0x7f697c158ca0 2023-12-16T21:57:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69637fe640 1 -- 172.21.15.28:0/3929778185 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69740f03c0 con 0x7f697c158ca0 2023-12-16T21:57:53.408 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6968066470 msgr2=0x7f6968068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6968066470 0x7f6968068930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f696c000f70 tx=0x7f696c009040 comp rx=0 tx=0).stop 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c158ca0 msgr2=0x7f697c15d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c158ca0 0x7f697c15d010 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f697401a580 tx=0x7f69740196f0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 shutdown_connections 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6968066470 0x7f6968068930 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f697c158ca0 0x7f697c15d010 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f697c0cba10 0x7f697c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 --2- 172.21.15.28:0/3929778185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f697c0b42d0 0x7f697c14cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 >> 172.21.15.28:0/3929778185 conn(0x7f697c0b0cc0 msgr2=0x7f697c152df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 shutdown_connections 2023-12-16T21:57:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.407+0000 7f69811bc640 1 -- 172.21.15.28:0/3929778185 wait complete. 2023-12-16T21:57:53.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:57:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.555+0000 7f2b9d58b640 1 Processor -- start 2023-12-16T21:57:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.555+0000 7f2b9d58b640 1 -- start start 2023-12-16T21:57:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b9d58b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b9d58b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b98146df0 0x7f2b9806c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b9d58b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b98161e60 con 0x7f2b980cdad0 2023-12-16T21:57:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b9d58b640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b981616a0 con 0x7f2b98146df0 2023-12-16T21:57:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b9d58b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9815db40 con 0x7f2b98146a10 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b977fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b98146df0 0x7f2b9806c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b977fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b98146df0 0x7f2b9806c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46354/0 (socket says 172.21.15.28:46354) 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b977fe640 1 -- 172.21.15.28:0/3438416721 learned_addr learned my addr 172.21.15.28:0/3438416721 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b96ffd640 1 --2- 172.21.15.28:0/3438416721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b977fe640 1 -- 172.21.15.28:0/3438416721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 msgr2=0x7f2b9815e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b977fe640 1 --2- 172.21.15.28:0/3438416721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b977fe640 1 -- 172.21.15.28:0/3438416721 >> v1:172.21.15.122:6789/0 conn(0x7f2b980cdad0 legacy=0x7f2b98160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b977fe640 1 -- 172.21.15.28:0/3438416721 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b981618a0 con 0x7f2b98146df0 2023-12-16T21:57:53.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b96ffd640 1 --2- 172.21.15.28:0/3438416721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:53.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b977fe640 1 --2- 172.21.15.28:0/3438416721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b9806c070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009870 tx=0x7f2b9001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18504eb91f8008b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b95ffb640 1 -- 172.21.15.28:0/3438416721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b900ac600 con 0x7f2b98146df0 2023-12-16T21:57:53.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b95ffb640 1 -- 172.21.15.28:0/3438416721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b900a8050 con 0x7f2b98146df0 2023-12-16T21:57:53.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.559+0000 7f2b95ffb640 1 -- 172.21.15.28:0/3438416721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b900ac980 con 0x7f2b98146df0 2023-12-16T21:57:53.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- 172.21.15.28:0/3438416721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 msgr2=0x7f2b9806c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/3438416721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b9806c070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009870 tx=0x7f2b9001c4d0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- 172.21.15.28:0/3438416721 shutdown_connections 2023-12-16T21:57:53.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/3438416721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b9806c070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/3438416721 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- 172.21.15.28:0/3438416721 >> 172.21.15.28:0/3438416721 conn(0x7f2b9806e810 msgr2=0x7f2b9806ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:53.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- 172.21.15.28:0/3438416721 shutdown_connections 2023-12-16T21:57:53.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- 172.21.15.28:0/3438416721 wait complete. 2023-12-16T21:57:53.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 Processor -- start 2023-12-16T21:57:53.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- start start 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b980cdad0 0x7f2b9813ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b980b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9813f160 con 0x7f2b98146df0 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9815c8a0 con 0x7f2b980cdad0 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b9d58b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b98143aa0 con 0x7f2b98146a10 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b980b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b980b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46364/0 (socket says 172.21.15.28:46364) 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b977fe640 1 -- 172.21.15.28:0/1671521267 learned_addr learned my addr 172.21.15.28:0/1671521267 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b967fc640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b96ffd640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b980cdad0 0x7f2b9813ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b977fe640 1 -- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 msgr2=0x7f2b9815c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b977fe640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b977fe640 1 -- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b980cdad0 msgr2=0x7f2b9813ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b977fe640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b980cdad0 0x7f2b9813ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b977fe640 1 -- 172.21.15.28:0/1671521267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b900094d0 con 0x7f2b98146df0 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.563+0000 7f2b967fc640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b977fe640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b980b3d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2b9001cd70 tx=0x7f2b90019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b8f7fe640 1 -- 172.21.15.28:0/1671521267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b900b5610 con 0x7f2b98146df0 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b8f7fe640 1 -- 172.21.15.28:0/1671521267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b900b5790 con 0x7f2b98146df0 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b8f7fe640 1 -- 172.21.15.28:0/1671521267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b900b5b40 con 0x7f2b98146df0 2023-12-16T21:57:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b98143d80 con 0x7f2b98146df0 2023-12-16T21:57:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b98144340 con 0x7f2b98146df0 2023-12-16T21:57:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b98144530 con 0x7f2b98146df0 2023-12-16T21:57:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b8f7fe640 1 -- 172.21.15.28:0/1671521267 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f2b900b3040 con 0x7f2b98146df0 2023-12-16T21:57:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b8f7fe640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2b78066470 0x7f2b78068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b8f7fe640 1 -- 172.21.15.28:0/1671521267 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b9011df70 con 0x7f2b98146df0 2023-12-16T21:57:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b96ffd640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2b78066470 0x7f2b78068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.567+0000 7f2b96ffd640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2b78066470 0x7f2b78068930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2b98067c90 tx=0x7f2b88008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.571+0000 7f2b8f7fe640 1 -- 172.21.15.28:0/1671521267 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f2b900eb480 con 0x7f2b98146df0 2023-12-16T21:57:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b980674d0 con 0x7f2b98146df0 2023-12-16T21:57:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b8f7fe640 1 -- 172.21.15.28:0/1671521267 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2b900eeaa0 con 0x7f2b98146df0 2023-12-16T21:57:53.732 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:57:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2b78066470 msgr2=0x7f2b78068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2b78066470 0x7f2b78068930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2b98067c90 tx=0x7f2b88008040 comp rx=0 tx=0).stop 2023-12-16T21:57:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 msgr2=0x7f2b980b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b980b3d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2b9001cd70 tx=0x7f2b90019270 comp rx=0 tx=0).stop 2023-12-16T21:57:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 shutdown_connections 2023-12-16T21:57:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2b78066470 0x7f2b78068930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b98146df0 0x7f2b980b3d00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2b98146a10 0x7f2b9815c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 --2- 172.21.15.28:0/1671521267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2b980cdad0 0x7f2b9813ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 >> 172.21.15.28:0/1671521267 conn(0x7f2b9806e810 msgr2=0x7f2b9815eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 shutdown_connections 2023-12-16T21:57:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.731+0000 7f2b9d58b640 1 -- 172.21.15.28:0/1671521267 wait complete. 2023-12-16T21:57:53.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:57:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f8652f640 1 Processor -- start 2023-12-16T21:57:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f8652f640 1 -- start start 2023-12-16T21:57:53.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f8652f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f8013cf30 0x7f6f8006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f8652f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f6f8015ad40 0x7f6f800cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f8652f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8015b3a0 con 0x7f6f8006c800 2023-12-16T21:57:53.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f8652f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f8013d310 con 0x7f6f8015ad40 2023-12-16T21:57:53.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f8652f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8015b1b0 con 0x7f6f8013cf30 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7f7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6f8006c800 0x7f6f8015a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51306/0 (socket says 172.21.15.28:51306) 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7f7fe640 1 -- 172.21.15.28:0/4136661978 learned_addr learned my addr 172.21.15.28:0/4136661978 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f84acc640 1 --2- 172.21.15.28:0/4136661978 >> v2:172.21.15.122:3300/0 conn(0x7f6f8015ad40 0x7f6f800cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7ffff640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f8013cf30 0x7f6f8006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059417884 0 0) 0x7f6f8015b3a0 con 0x7f6f8006c800 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f68003530 con 0x7f6f8006c800 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3987907270 0 0) 0x7f6f68003530 con 0x7f6f8006c800 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f8013cf30 msgr2=0x7f6f8006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f8013cf30 0x7f6f8006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 >> v2:172.21.15.122:3300/0 conn(0x7f6f8015ad40 msgr2=0x7f6f800cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 --2- 172.21.15.28:0/4136661978 >> v2:172.21.15.122:3300/0 conn(0x7f6f8015ad40 0x7f6f800cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8006cc40 con 0x7f6f8006c800 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f6f74001680 con 0x7f6f8006c800 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 >> v1:172.21.15.28:6789/0 conn(0x7f6f8006c800 legacy=0x7f6f8015a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f8013cf30 0x7f6f68003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f68003f00 0x7f6f680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f68004d10 0x7f6f68005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6800b990 con 0x7f6f68004d10 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6800bad0 con 0x7f6f68003f00 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6800bc60 con 0x7f6f8013cf30 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f84acc640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f68004d10 0x7f6f68005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7f7fe640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f68003f00 0x7f6f680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f7ffff640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f8013cf30 0x7f6f68003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f84acc640 1 -- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f8013cf30 msgr2=0x7f6f68003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f84acc640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f8013cf30 0x7f6f68003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f84acc640 1 -- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f68003f00 msgr2=0x7f6f680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f84acc640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f68003f00 0x7f6f680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.867+0000 7f6f84acc640 1 -- 172.21.15.28:0/4136661978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f68003530 con 0x7f6f68004d10 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f84acc640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f68004d10 0x7f6f68005190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009bd0 tx=0x7f6f7801ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8610cba459121e77 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f780a7ea0 con 0x7f6f68004d10 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7800f040 con 0x7f6f68004d10 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f7effd640 1 -- 172.21.15.28:0/4136661978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f780af680 con 0x7f6f68004d10 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 -- 172.21.15.28:0/4136661978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f68004d10 msgr2=0x7f6f68005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 --2- 172.21.15.28:0/4136661978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f68004d10 0x7f6f68005190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6f78009bd0 tx=0x7f6f7801ee90 comp rx=0 tx=0).stop 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f84acc640 1 -- 172.21.15.28:0/4136661978 reap_dead start 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 -- 172.21.15.28:0/4136661978 shutdown_connections 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 -- 172.21.15.28:0/4136661978 >> 172.21.15.28:0/4136661978 conn(0x7f6f8006a530 msgr2=0x7f6f8006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 -- 172.21.15.28:0/4136661978 shutdown_connections 2023-12-16T21:57:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 -- 172.21.15.28:0/4136661978 wait complete. 2023-12-16T21:57:53.879 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 Processor -- start 2023-12-16T21:57:53.879 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 -- start start 2023-12-16T21:57:53.879 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f68004d10 0x7f6f8013b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.879 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f80154b30 0x7f6f800b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8013e360 0x7f6f8013e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8013ed20 con 0x7f6f8013e360 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.871+0000 7f6f8652f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8013b910 con 0x7f6f80154b30 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f8652f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f800b3cb0 con 0x7f6f68004d10 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8013e360 0x7f6f8013e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8013e360 0x7f6f8013e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46374/0 (socket says 172.21.15.28:46374) 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 -- 172.21.15.28:0/3599638219 learned_addr learned my addr 172.21.15.28:0/3599638219 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7ffff640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f68004d10 0x7f6f8013b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7f7fe640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f80154b30 0x7f6f800b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 -- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f68004d10 msgr2=0x7f6f8013b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f68004d10 0x7f6f8013b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 -- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f80154b30 msgr2=0x7f6f800b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f80154b30 0x7f6f800b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:53.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 -- 172.21.15.28:0/3599638219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f801550d0 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f84acc640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8013e360 0x7f6f8013e7e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6f780b2040 tx=0x7f6f78009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7cff9640 1 -- 172.21.15.28:0/3599638219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f780b0b80 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7cff9640 1 -- 172.21.15.28:0/3599638219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f780b0ce0 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f801553b0 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7cff9640 1 -- 172.21.15.28:0/3599638219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78003be0 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f80155970 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f80069440 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7cff9640 1 -- 172.21.15.28:0/3599638219 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6f78003d40 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7cff9640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6f6c0664a0 0x7f6f6c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7cff9640 1 -- 172.21.15.28:0/3599638219 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f7811e460 con 0x7f6f8013e360 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7ffff640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6f6c0664a0 0x7f6f6c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:53.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.875+0000 7f6f7ffff640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6f6c0664a0 0x7f6f6c068960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009b20 tx=0x7f6f70008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:53.882 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:53.879+0000 7f6f7cff9640 1 -- 172.21.15.28:0/3599638219 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6f780eb900 con 0x7f6f8013e360 2023-12-16T21:57:54.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.067+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f800cc300 con 0x7f6f8013e360 2023-12-16T21:57:54.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.067+0000 7f6f7cff9640 1 -- 172.21.15.28:0/3599638219 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4932 (secure 0 0 0) 0x7f6f780be090 con 0x7f6f8013e360 2023-12-16T21:57:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:54.070 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:57:42.850798+0000","modified":"2023-12-16T21:57:45.892479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:57:54.070 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2023-12-16T21:57:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6f6c0664a0 msgr2=0x7f6f6c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6f6c0664a0 0x7f6f6c068960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009b20 tx=0x7f6f70008040 comp rx=0 tx=0).stop 2023-12-16T21:57:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8013e360 msgr2=0x7f6f8013e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8013e360 0x7f6f8013e7e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6f780b2040 tx=0x7f6f78009750 comp rx=0 tx=0).stop 2023-12-16T21:57:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 shutdown_connections 2023-12-16T21:57:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6f6c0664a0 0x7f6f6c068960 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8013e360 0x7f6f8013e7e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f80154b30 0x7f6f800b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 --2- 172.21.15.28:0/3599638219 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6f68004d10 0x7f6f8013b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 >> 172.21.15.28:0/3599638219 conn(0x7f6f8006a530 msgr2=0x7f6f8015cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:54.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 shutdown_connections 2023-12-16T21:57:54.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.071+0000 7f6f8652f640 1 -- 172.21.15.28:0/3599638219 wait complete. 2023-12-16T21:57:54.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:57:54.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.195+0000 7f5a2994f640 1 Processor -- start 2023-12-16T21:57:54.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.195+0000 7f5a2994f640 1 -- start start 2023-12-16T21:57:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a2994f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a2406c2c0 0x7f5a2406c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a2994f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5a240b3400 0x7f5a240b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a2994f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a241408c0 con 0x7f5a24140180 2023-12-16T21:57:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a2994f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2406ccb0 con 0x7f5a240b3400 2023-12-16T21:57:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a2994f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2415db80 con 0x7f5a2406c2c0 2023-12-16T21:57:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a237fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5a240b3400 0x7f5a240b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a2406c2c0 0x7f5a2406c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a22ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5a24140180 0x7f5a24071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:58956/0 (socket says 172.21.15.28:58956) 2023-12-16T21:57:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a22ffd640 1 -- 172.21.15.28:0/2587060790 learned_addr learned my addr 172.21.15.28:0/2587060790 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a237fe640 1 -- 172.21.15.28:0/2587060790 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a2406c2c0 msgr2=0x7f5a2406c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a237fe640 1 --2- 172.21.15.28:0/2587060790 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a2406c2c0 0x7f5a2406c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a237fe640 1 -- 172.21.15.28:0/2587060790 >> v1:172.21.15.122:6789/0 conn(0x7f5a24140180 legacy=0x7f5a24071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a237fe640 1 -- 172.21.15.28:0/2587060790 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2415de60 con 0x7f5a240b3400 2023-12-16T21:57:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a237fe640 1 --2- 172.21.15.28:0/2587060790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a240b3400 0x7f5a240b3830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009d90 tx=0x7f5a1c01c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266772ca89a578ba server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a227fc640 1 -- 172.21.15.28:0/2587060790 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a240b4580 con 0x7f5a240b3400 2023-12-16T21:57:54.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a227fc640 1 -- 172.21.15.28:0/2587060790 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1c0a5950 con 0x7f5a240b3400 2023-12-16T21:57:54.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.199+0000 7f5a227fc640 1 -- 172.21.15.28:0/2587060790 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1c0a5ce0 con 0x7f5a240b3400 2023-12-16T21:57:54.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- 172.21.15.28:0/2587060790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a240b3400 msgr2=0x7f5a240b3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 --2- 172.21.15.28:0/2587060790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a240b3400 0x7f5a240b3830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009d90 tx=0x7f5a1c01c890 comp rx=0 tx=0).stop 2023-12-16T21:57:54.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- 172.21.15.28:0/2587060790 shutdown_connections 2023-12-16T21:57:54.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 --2- 172.21.15.28:0/2587060790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a240b3400 0x7f5a240b3830 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 --2- 172.21.15.28:0/2587060790 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a2406c2c0 0x7f5a2406c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- 172.21.15.28:0/2587060790 >> 172.21.15.28:0/2587060790 conn(0x7f5a2406a530 msgr2=0x7f5a2406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:54.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- 172.21.15.28:0/2587060790 shutdown_connections 2023-12-16T21:57:54.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- 172.21.15.28:0/2587060790 wait complete. 2023-12-16T21:57:54.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 Processor -- start 2023-12-16T21:57:54.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- start start 2023-12-16T21:57:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a2406c2c0 0x7f5a24139390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a240b3400 0x7f5a2413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a24140180 0x7f5a240aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241385d0 con 0x7f5a240b3400 2023-12-16T21:57:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241388b0 con 0x7f5a2406c2c0 2023-12-16T21:57:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a2994f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a24138b90 con 0x7f5a24140180 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a240b3400 0x7f5a2413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a240b3400 0x7f5a2413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46376/0 (socket says 172.21.15.28:46376) 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 -- 172.21.15.28:0/3818165134 learned_addr learned my addr 172.21.15.28:0/3818165134 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a237fe640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a24140180 0x7f5a240aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a22ffd640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a2406c2c0 0x7f5a24139390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 -- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a24140180 msgr2=0x7f5a240aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a24140180 0x7f5a240aec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 -- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a2406c2c0 msgr2=0x7f5a24139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a2406c2c0 0x7f5a24139390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 -- 172.21.15.28:0/3818165134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c009a70 con 0x7f5a240b3400 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a237fe640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a24140180 0x7f5a240aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a22ffd640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a2406c2c0 0x7f5a24139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.203+0000 7f5a1bfff640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a240b3400 0x7f5a2413ed90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00e930 tx=0x7f5a0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:54.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a20ff9640 1 -- 172.21.15.28:0/3818165134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c009690 con 0x7f5a240b3400 2023-12-16T21:57:54.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a20ff9640 1 -- 172.21.15.28:0/3818165134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c00ca00 con 0x7f5a240b3400 2023-12-16T21:57:54.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a24138e70 con 0x7f5a240b3400 2023-12-16T21:57:54.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a20ff9640 1 -- 172.21.15.28:0/3818165134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c00cd70 con 0x7f5a240b3400 2023-12-16T21:57:54.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a240af5b0 con 0x7f5a240b3400 2023-12-16T21:57:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a24069440 con 0x7f5a240b3400 2023-12-16T21:57:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a20ff9640 1 -- 172.21.15.28:0/3818165134 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f5a0c013470 con 0x7f5a240b3400 2023-12-16T21:57:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a20ff9640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5a04066470 0x7f5a04068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a20ff9640 1 -- 172.21.15.28:0/3818165134 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a0c084990 con 0x7f5a240b3400 2023-12-16T21:57:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a22ffd640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5a04066470 0x7f5a04068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.207+0000 7f5a22ffd640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5a04066470 0x7f5a04068930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5a24157370 tx=0x7f5a14009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:54.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.211+0000 7f5a20ff9640 1 -- 172.21.15.28:0/3818165134 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f5a0c051ec0 con 0x7f5a240b3400 2023-12-16T21:57:54.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.363+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a24138710 con 0x7f5a240b3400 2023-12-16T21:57:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.363+0000 7f5a20ff9640 1 -- 172.21.15.28:0/3818165134 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f5a24138710 con 0x7f5a240b3400 2023-12-16T21:57:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:54.367 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:57:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5a04066470 msgr2=0x7f5a04068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5a04066470 0x7f5a04068930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5a24157370 tx=0x7f5a14009040 comp rx=0 tx=0).stop 2023-12-16T21:57:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a240b3400 msgr2=0x7f5a2413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a240b3400 0x7f5a2413ed90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00e930 tx=0x7f5a0c00edf0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 shutdown_connections 2023-12-16T21:57:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5a04066470 0x7f5a04068930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5a24140180 0x7f5a240aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a240b3400 0x7f5a2413ed90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 --2- 172.21.15.28:0/3818165134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a2406c2c0 0x7f5a24139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 >> 172.21.15.28:0/3818165134 conn(0x7f5a2406a530 msgr2=0x7f5a240afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 shutdown_connections 2023-12-16T21:57:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.367+0000 7f5a2994f640 1 -- 172.21.15.28:0/3818165134 wait complete. 2023-12-16T21:57:54.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:57:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc474a640 1 Processor -- start 2023-12-16T21:57:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc474a640 1 -- start start 2023-12-16T21:57:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc474a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f8fbc15ad40 0x7f8fbc164df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc474a640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc15b170 con 0x7f8fbc13cf30 2023-12-16T21:57:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc474a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc15b3a0 con 0x7f8fbc06c9d0 2023-12-16T21:57:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc474a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc13d310 con 0x7f8fbc15ad40 2023-12-16T21:57:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc24e6640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8fbc13cf30 0x7f8fbc06c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51310/0 (socket says 172.21.15.28:51310) 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc24e6640 1 -- 172.21.15.28:0/923733443 learned_addr learned my addr 172.21.15.28:0/923733443 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc2ce7640 1 --2- 172.21.15.28:0/923733443 >> v2:172.21.15.122:3300/0 conn(0x7f8fbc15ad40 0x7f8fbc164df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901883428 0 0) 0x7f8fbc15b170 con 0x7f8fbc13cf30 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa4005740 con 0x7f8fbc13cf30 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2982314964 0 0) 0x7f8fa4005740 con 0x7f8fbc13cf30 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 >> v2:172.21.15.122:3300/0 conn(0x7f8fbc15ad40 msgr2=0x7f8fbc164df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 --2- 172.21.15.28:0/923733443 >> v2:172.21.15.122:3300/0 conn(0x7f8fbc15ad40 0x7f8fbc164df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 >> v1:172.21.15.122:6789/0 conn(0x7f8fbc06c9d0 legacy=0x7f8fbc15a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc13f720 con 0x7f8fbc13cf30 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f8fac001680 con 0x7f8fbc13cf30 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 >> v1:172.21.15.28:6789/0 conn(0x7f8fbc13cf30 legacy=0x7f8fbc06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa4005b90 0x7f8fa4005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa40069e0 0x7f8fa4006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fa40073a0 0x7f8fa400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa400e310 con 0x7f8fa4005b90 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa400e480 con 0x7f8fa40069e0 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa400e640 con 0x7f8fa40073a0 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc24e6640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa4005b90 0x7f8fa4005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc1ce5640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa40069e0 0x7f8fa4006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc2ce7640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fa40073a0 0x7f8fa400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc24e6640 1 -- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fa40073a0 msgr2=0x7f8fa400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc24e6640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fa40073a0 0x7f8fa400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc24e6640 1 -- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa40069e0 msgr2=0x7f8fa4006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc24e6640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa40069e0 0x7f8fa4006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc24e6640 1 -- 172.21.15.28:0/923733443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa4005740 con 0x7f8fa4005b90 2023-12-16T21:57:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc2ce7640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fa40073a0 0x7f8fa400dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:54.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.503+0000 7f8fc2ce7640 1 -- 172.21.15.28:0/923733443 reap_dead start 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc24e6640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa4005b90 0x7f8fa4005fb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8fac001060 tx=0x7f8fac01cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d13a11e934c7eb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac0058e0 con 0x7f8fa4005b90 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fac005a40 con 0x7f8fa4005b90 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc14e4640 1 -- 172.21.15.28:0/923733443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac005e70 con 0x7f8fa4005b90 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 -- 172.21.15.28:0/923733443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa4005b90 msgr2=0x7f8fa4005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa4005b90 0x7f8fa4005fb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8fac001060 tx=0x7f8fac01cb40 comp rx=0 tx=0).stop 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 -- 172.21.15.28:0/923733443 shutdown_connections 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 --2- 172.21.15.28:0/923733443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa4005b90 0x7f8fa4005fb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 -- 172.21.15.28:0/923733443 >> 172.21.15.28:0/923733443 conn(0x7f8fbc06a530 msgr2=0x7f8fbc06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 -- 172.21.15.28:0/923733443 shutdown_connections 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 -- 172.21.15.28:0/923733443 wait complete. 2023-12-16T21:57:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 Processor -- start 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.507+0000 7f8fc474a640 1 -- start start 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa4005b90 0x7f8fbc1f1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fbc136c80 0x7f8fbc1f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ce20 0x7f8fbc0b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc135c50 con 0x7f8fbc15ce20 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc1f5de0 con 0x7f8fa4005b90 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc135ec0 con 0x7f8fbc136c80 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ce20 0x7f8fbc0b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ce20 0x7f8fbc0b2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46398/0 (socket says 172.21.15.28:46398) 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 -- 172.21.15.28:0/2238356030 learned_addr learned my addr 172.21.15.28:0/2238356030 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc24e6640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa4005b90 0x7f8fbc1f1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc1ce5640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fbc136c80 0x7f8fbc1f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 -- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fbc136c80 msgr2=0x7f8fbc1f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fbc136c80 0x7f8fbc1f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 -- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa4005b90 msgr2=0x7f8fbc1f1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa4005b90 0x7f8fbc1f1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 -- 172.21.15.28:0/2238356030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc1361a0 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc1ce5640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fbc136c80 0x7f8fbc1f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc2ce7640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ce20 0x7f8fbc0b2d70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc06cf10 tx=0x7f8fb400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fbaffd640 1 -- 172.21.15.28:0/2238356030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4016620 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fbaffd640 1 -- 172.21.15.28:0/2238356030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb4016780 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc136480 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fbaffd640 1 -- 172.21.15.28:0/2238356030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4016b10 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc0b32b0 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fbc069440 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fbaffd640 1 -- 172.21.15.28:0/2238356030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f8fb4016c70 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fbaffd640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8fa80664a0 0x7f8fa8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fbaffd640 1 -- 172.21.15.28:0/2238356030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fb4084cc0 con 0x7f8fbc15ce20 2023-12-16T21:57:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc24e6640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8fa80664a0 0x7f8fa8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.511+0000 7f8fc24e6640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8fa80664a0 0x7f8fa8068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0aa000 tx=0x7f8fac001a10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.515+0000 7f8fbaffd640 1 -- 172.21.15.28:0/2238356030 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f8fb4052140 con 0x7f8fbc15ce20 2023-12-16T21:57:54.708 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.707+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fbc136930 con 0x7f8fbc15ce20 2023-12-16T21:57:54.709 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.707+0000 7f8fbaffd640 1 -- 172.21.15.28:0/2238356030 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4932 (secure 0 0 0) 0x7f8fb4055760 con 0x7f8fbc15ce20 2023-12-16T21:57:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:57:54.710 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:57:42.850798+0000","modified":"2023-12-16T21:57:45.892479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:57:54.710 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8fa80664a0 msgr2=0x7f8fa8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8fa80664a0 0x7f8fa8068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0aa000 tx=0x7f8fac001a10 comp rx=0 tx=0).stop 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ce20 msgr2=0x7f8fbc0b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ce20 0x7f8fbc0b2d70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc06cf10 tx=0x7f8fb400f520 comp rx=0 tx=0).stop 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 shutdown_connections 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8fa80664a0 0x7f8fa8068960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fbc15ce20 0x7f8fbc0b2d70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8fbc136c80 0x7f8fbc1f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 --2- 172.21.15.28:0/2238356030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fa4005b90 0x7f8fbc1f1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 >> 172.21.15.28:0/2238356030 conn(0x7f8fbc06a530 msgr2=0x7f8fbc138e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 shutdown_connections 2023-12-16T21:57:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:54.711+0000 7f8fc474a640 1 -- 172.21.15.28:0/2238356030 wait complete. 2023-12-16T21:57:55.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-16T21:57:42.850798+0000', 'modified': '2023-12-16T21:57:45.892479+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4285': {'gid': 4285, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6835/771354521', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 771354521}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 771354521}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-16T21:57:55.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-16T21:57:55.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-12-16T21:57:55.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d5d8d0640 1 Processor -- start 2023-12-16T21:57:55.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d5d8d0640 1 -- start start 2023-12-16T21:57:55.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d5d8d0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d580b3bc0 0x7f4d580b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d5d8d0640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 0x7f4d580cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d5d8d0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d580d0d30 con 0x7f4d580b2f40 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d5d8d0640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d58160f80 con 0x7f4d580b3bc0 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d5d8d0640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d580d12f0 con 0x7f4d580d1e90 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d56ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d580b3bc0 0x7f4d580b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d56ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d580b3bc0 0x7f4d580b2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46402/0 (socket says 172.21.15.28:46402) 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d56ffd640 1 -- 172.21.15.28:0/4270524149 learned_addr learned my addr 172.21.15.28:0/4270524149 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d577fe640 1 --2- 172.21.15.28:0/4270524149 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 0x7f4d580cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d56ffd640 1 -- 172.21.15.28:0/4270524149 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 msgr2=0x7f4d580cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/4270524149 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 0x7f4d580cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d56ffd640 1 -- 172.21.15.28:0/4270524149 >> v1:172.21.15.122:6789/0 conn(0x7f4d580b2f40 legacy=0x7f4d580d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.267+0000 7f4d56ffd640 1 -- 172.21.15.28:0/4270524149 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d580cecc0 con 0x7f4d580b3bc0 2023-12-16T21:57:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/4270524149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d580b2a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009870 tx=0x7f4d4801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c570f4f5600388d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d55ffb640 1 -- 172.21.15.28:0/4270524149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d480ac600 con 0x7f4d580b3bc0 2023-12-16T21:57:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d55ffb640 1 -- 172.21.15.28:0/4270524149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d480a8050 con 0x7f4d580b3bc0 2023-12-16T21:57:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d55ffb640 1 -- 172.21.15.28:0/4270524149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d480ac980 con 0x7f4d580b3bc0 2023-12-16T21:57:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/4270524149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 msgr2=0x7f4d580b2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/4270524149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d580b2a00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009870 tx=0x7f4d4801c4d0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/4270524149 shutdown_connections 2023-12-16T21:57:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/4270524149 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 0x7f4d580cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/4270524149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d580b2a00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/4270524149 >> 172.21.15.28:0/4270524149 conn(0x7f4d5809c430 msgr2=0x7f4d580d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/4270524149 shutdown_connections 2023-12-16T21:57:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/4270524149 wait complete. 2023-12-16T21:57:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 Processor -- start 2023-12-16T21:57:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.271+0000 7f4d5d8d0640 1 -- start start 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d5d8d0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d580b2f40 0x7f4d5814e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d5d8d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d5815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d5d8d0640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 0x7f4d5815f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d5d8d0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5815d580 con 0x7f4d580b3bc0 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d5d8d0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d581f6640 con 0x7f4d580b2f40 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d5d8d0640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d581f6920 con 0x7f4d580d1e90 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d5815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d5815cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46414/0 (socket says 172.21.15.28:46414) 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 -- 172.21.15.28:0/504994835 learned_addr learned my addr 172.21.15.28:0/504994835 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d580b2f40 0x7f4d5814e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d577fe640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 0x7f4d5815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 -- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 msgr2=0x7f4d5815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 0x7f4d5815f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 -- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d580b2f40 msgr2=0x7f4d5814e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d580b2f40 0x7f4d5814e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 -- 172.21.15.28:0/504994835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d480094d0 con 0x7f4d580b3bc0 2023-12-16T21:57:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d567fc640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d5815cae0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00c410 tx=0x7f4d4c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d3ffff640 1 -- 172.21.15.28:0/504994835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c011790 con 0x7f4d580b3bc0 2023-12-16T21:57:55.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d3ffff640 1 -- 172.21.15.28:0/504994835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c0118f0 con 0x7f4d580b3bc0 2023-12-16T21:57:55.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/504994835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d581f6c00 con 0x7f4d580b3bc0 2023-12-16T21:57:55.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d3ffff640 1 -- 172.21.15.28:0/504994835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c011ce0 con 0x7f4d580b3bc0 2023-12-16T21:57:55.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/504994835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5814ec70 con 0x7f4d580b3bc0 2023-12-16T21:57:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d3ffff640 1 -- 172.21.15.28:0/504994835 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4d4c01c460 con 0x7f4d580b3bc0 2023-12-16T21:57:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d3ffff640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4d44066470 0x7f4d44068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d3ffff640 1 -- 172.21.15.28:0/504994835 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d4c083e40 con 0x7f4d580b3bc0 2023-12-16T21:57:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.275+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4d44066470 0x7f4d44068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 Processor -- start 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- start start 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4d44066470 0x7f4d44068930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4d480ae000 tx=0x7f4d480a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454040 0x557a10454420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454960 0x557a1040a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a1040aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a102d1f00 con 0x557a1040a690 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a10261dd0 con 0x557a10454040 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a1025f300 con 0x557a10454960 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a1040aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a1040aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46422/0 (socket says 172.21.15.28:46422) 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 -- 172.21.15.28:0/3743178878 learned_addr learned my addr 172.21.15.28:0/3743178878 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454040 0x557a10454420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d567fc640 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454960 0x557a1040a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 -- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454960 msgr2=0x557a1040a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454960 0x557a1040a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 -- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454040 msgr2=0x557a10454420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454040 0x557a10454420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 -- 172.21.15.28:0/3743178878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a0ff52890 con 0x557a1040a690 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d567fc640 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454960 0x557a1040a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d577fe640 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a1040aa70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009fc0 tx=0x7f4d500077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b486ea0738c7988 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d3d7fa640 1 -- 172.21.15.28:0/3743178878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d500147e0 con 0x557a1040a690 2023-12-16T21:57:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d3d7fa640 1 -- 172.21.15.28:0/3743178878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5000f050 con 0x557a1040a690 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d3d7fa640 1 -- 172.21.15.28:0/3743178878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50014b60 con 0x557a1040a690 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- 172.21.15.28:0/3743178878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 msgr2=0x557a1040aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a1040aa70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009fc0 tx=0x7f4d500077c0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- 172.21.15.28:0/3743178878 shutdown_connections 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a1040aa70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454960 0x557a1040a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 --2- 172.21.15.28:0/3743178878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454040 0x557a10454420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- 172.21.15.28:0/3743178878 >> 172.21.15.28:0/3743178878 conn(0x557a10406640 msgr2=0x557a10408a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- 172.21.15.28:0/3743178878 shutdown_connections 2023-12-16T21:57:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- 172.21.15.28:0/3743178878 wait complete. 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 -- start start 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.279+0000 7f4d60a01000 1 Processor -- start 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a10415c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 0x557a10416170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104166b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a1025abf0 con 0x557a1040a690 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a10212c60 con 0x557a10454960 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a10218b00 con 0x557a10454040 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 10 client.0 _ll_register_callbacks cb 0x557a103f5710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a10415c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a10415c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46434/0 (socket says 172.21.15.28:46434) 2023-12-16T21:57:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 -- 172.21.15.28:0/2386291556 learned_addr learned my addr 172.21.15.28:0/2386291556 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d577fe640 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d567fc640 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 0x557a10416170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 -- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 msgr2=0x557a10416170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 0x557a10416170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 -- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 msgr2=0x557a104166b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104166b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 -- 172.21.15.28:0/2386291556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d50009bf0 con 0x557a1040a690 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d577fe640 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104166b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a10415c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4d4801cd70 tx=0x7f4d4801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb9fd01183bab9a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d367fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d367fc640 1 -- 172.21.15.28:0/2386291556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d480cc680 con 0x557a1040a690 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d367fc640 1 -- 172.21.15.28:0/2386291556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d480c7070 con 0x557a1040a690 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 -- 172.21.15.28:0/2386291556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a10218490 con 0x557a1040a690 2023-12-16T21:57:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d367fc640 1 -- 172.21.15.28:0/2386291556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d480b6440 con 0x557a1040a690 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 10 client.4614 fetch_fsmap learned FSMap version 8 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 -- 172.21.15.28:0/2386291556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x557a1021b200 con 0x557a1040a690 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d367fc640 1 -- 172.21.15.28:0/2386291556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557a10218490 con 0x557a1040a690 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d367fc640 1 -- 172.21.15.28:0/2386291556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4d480be040 con 0x557a1040a690 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 10 client.4614 fetch_fsmap finished waiting for FSMap version 8 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 10 client.4614 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 20 client.4614 populate_metadata read hostname 'smithi028' 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] conn(0x557a104640e0 0x557a104664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 4 client.4614 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d60a01000 1 -- 172.21.15.28:0/2386291556 --> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a10259540 con 0x557a104640e0 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d567fc640 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] conn(0x557a104640e0 0x557a104664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d567fc640 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] conn(0x557a104640e0 0x557a104664d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf7d907e9b3f9353 server_cookie=162118f6d557f4c0 in_seq=0 out_seq=0 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.283+0000 7f4d367fc640 10 client.4614 ms_handle_connect on v2:172.21.15.122:6834/771354521 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d367fc640 1 -- 172.21.15.28:0/2386291556 <== mds.0 v2:172.21.15.122:6834/771354521 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557a10259540 con 0x557a104640e0 2023-12-16T21:57:55.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d367fc640 10 client.4614 handle_command_reply: tid=0 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 client.4614 shutdown 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 -- 172.21.15.28:0/2386291556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 msgr2=0x557a10415c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a10415c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4d4801cd70 tx=0x7f4d4801ad10 comp rx=0 tx=0).stop 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 -- 172.21.15.28:0/2386291556 shutdown_connections 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104166b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 0x557a10416170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] conn(0x557a104640e0 0x557a104664d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 --2- 172.21.15.28:0/2386291556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a1040a690 0x557a10415c30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d367fc640 0 client.4614 ms_handle_reset on v2:172.21.15.122:6834/771354521 2023-12-16T21:57:55.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 -- 172.21.15.28:0/2386291556 >> 172.21.15.28:0/2386291556 conn(0x557a10406640 msgr2=0x557a104092c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 -- 172.21.15.28:0/2386291556 shutdown_connections 2023-12-16T21:57:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 1 -- 172.21.15.28:0/2386291556 wait complete. 2023-12-16T21:57:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.287+0000 7f4d60a01000 20 client.4614 trim_cache size 0 max 16384 2023-12-16T21:57:55.303 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 Processor -- start 2023-12-16T21:57:55.303 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- start start 2023-12-16T21:57:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a1040a690 0x557a10409a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 0x557a10409f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454960 0x557a104664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a10406500 con 0x557a10454960 2023-12-16T21:57:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a0ff13940 con 0x557a1040a690 2023-12-16T21:57:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a10401ac0 con 0x557a10454040 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454960 0x557a104664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454960 0x557a104664d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46448/0 (socket says 172.21.15.28:46448) 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 -- 172.21.15.28:0/474027048 learned_addr learned my addr 172.21.15.28:0/474027048 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 -- 172.21.15.28:0/474027048 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 msgr2=0x557a10409f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/474027048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a1040a690 0x557a10409a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 --2- 172.21.15.28:0/474027048 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 0x557a10409f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 -- 172.21.15.28:0/474027048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a1040a690 msgr2=0x557a10409a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 --2- 172.21.15.28:0/474027048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a1040a690 0x557a10409a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 -- 172.21.15.28:0/474027048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a10201500 con 0x557a10454960 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d577fe640 1 --2- 172.21.15.28:0/474027048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454960 0x557a104664d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4d500098d0 tx=0x7f4d50014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88ad5fce3a51c9e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d3dffb640 1 -- 172.21.15.28:0/474027048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50007000 con 0x557a10454960 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d3dffb640 1 -- 172.21.15.28:0/474027048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d500145e0 con 0x557a10454960 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d3dffb640 1 -- 172.21.15.28:0/474027048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50014990 con 0x557a10454960 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- 172.21.15.28:0/474027048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454960 msgr2=0x557a104664d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- 172.21.15.28:0/474027048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454960 0x557a104664d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4d500098d0 tx=0x7f4d50014000 comp rx=0 tx=0).stop 2023-12-16T21:57:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- 172.21.15.28:0/474027048 shutdown_connections 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- 172.21.15.28:0/474027048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454960 0x557a104664d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- 172.21.15.28:0/474027048 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a10454040 0x557a10409f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- 172.21.15.28:0/474027048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a1040a690 0x557a10409a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- 172.21.15.28:0/474027048 >> 172.21.15.28:0/474027048 conn(0x557a10406640 msgr2=0x557a10415d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- 172.21.15.28:0/474027048 shutdown_connections 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- 172.21.15.28:0/474027048 wait complete. 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- start start 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 Processor -- start 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a1040a690 0x557a10468500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454040 0x557a10470fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a10453a60 con 0x557a10454040 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a1040e6c0 con 0x557a10454960 2023-12-16T21:57:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a1025a400 con 0x557a1040a690 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.303+0000 7f4d60a01000 10 client.0 _ll_register_callbacks cb 0x557a103d5f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454040 0x557a10470fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454040 0x557a10470fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46458/0 (socket says 172.21.15.28:46458) 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 -- 172.21.15.28:0/1668911039 learned_addr learned my addr 172.21.15.28:0/1668911039 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a1040a690 0x557a10468500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d577fe640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 -- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a1040a690 msgr2=0x557a10468500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a1040a690 0x557a10468500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 -- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 msgr2=0x557a104714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 -- 172.21.15.28:0/1668911039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a0ff52890 con 0x557a10454040 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a1040a690 0x557a10468500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d577fe640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d567fc640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454040 0x557a10470fa0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00a040 tx=0x7f4d4c054fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8282d1faab6e348 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 1 -- 172.21.15.28:0/1668911039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c053060 con 0x557a10454040 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 1 -- 172.21.15.28:0/1668911039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d4c057040 con 0x557a10454040 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 -- 172.21.15.28:0/1668911039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a1040cdf0 con 0x557a10454040 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 1 -- 172.21.15.28:0/1668911039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d4c05ccd0 con 0x557a10454040 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 10 client.4620 fetch_fsmap learned FSMap version 8 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 -- 172.21.15.28:0/1668911039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x557a0ff52cc0 con 0x557a10454040 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 1 -- 172.21.15.28:0/1668911039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557a1040cdf0 con 0x557a10454040 2023-12-16T21:57:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 1 -- 172.21.15.28:0/1668911039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4d4c068bb0 con 0x557a10454040 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 10 client.4620 fetch_fsmap finished waiting for FSMap version 8 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 10 client.4620 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 20 client.4620 populate_metadata read hostname 'smithi028' 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] conn(0x557a104b4a00 0x557a104b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 4 client.4620 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 -- 172.21.15.28:0/1668911039 --> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557a101e1a50 con 0x557a104b4a00 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] conn(0x557a104b4a00 0x557a104b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d56ffd640 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] conn(0x557a104b4a00 0x557a104b4de0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=779594d31c66c0e6 server_cookie=f87987b18394ded1 in_seq=0 out_seq=0 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 10 client.4620 ms_handle_connect on v2:172.21.15.122:6834/771354521 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 1 -- 172.21.15.28:0/1668911039 <== mds.0 v2:172.21.15.122:6834/771354521 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x557a101e1a50 con 0x557a104b4a00 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 10 client.4620 handle_command_reply: tid=0 2023-12-16T21:57:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 client.4620 shutdown 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 -- 172.21.15.28:0/1668911039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454040 msgr2=0x557a10470fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454040 0x557a10470fa0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00a040 tx=0x7f4d4c054fc0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 -- 172.21.15.28:0/1668911039 shutdown_connections 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557a10454960 0x557a104714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:6834/771354521,v1:172.21.15.122:6835/771354521] conn(0x557a104b4a00 0x557a104b4de0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557a10454040 0x557a10470fa0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 --2- 172.21.15.28:0/1668911039 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x557a1040a690 0x557a10468500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d367fc640 0 client.4620 ms_handle_reset on v2:172.21.15.122:6834/771354521 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 -- 172.21.15.28:0/1668911039 >> 172.21.15.28:0/1668911039 conn(0x557a10406640 msgr2=0x557a10416420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 -- 172.21.15.28:0/1668911039 shutdown_connections 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 1 -- 172.21.15.28:0/1668911039 wait complete. 2023-12-16T21:57:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d60a01000 20 client.4620 trim_cache size 0 max 16384 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "fa65dfdd-b227-4a96-8047-b713453c357b", 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4285, 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 11.792485940000001, 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 8, 2023-12-16T21:57:55.320 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 23, 2023-12-16T21:57:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 23, 2023-12-16T21:57:55.321 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 18.624585903 2023-12-16T21:57:55.321 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-16T21:57:55.321 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4d44066470 msgr2=0x7f4d44068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4d44066470 0x7f4d44068930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4d480ae000 tx=0x7f4d480a7000 comp rx=0 tx=0).stop 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/504994835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 msgr2=0x7f4d5815cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d5815cae0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00c410 tx=0x7f4d4c00c8d0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/504994835 shutdown_connections 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4d44066470 0x7f4d44068930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4d580d1e90 0x7f4d5815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d580b3bc0 0x7f4d5815cae0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 --2- 172.21.15.28:0/504994835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d580b2f40 0x7f4d5814e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/504994835 >> 172.21.15.28:0/504994835 conn(0x7f4d5809c430 msgr2=0x7f4d581606d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/504994835 shutdown_connections 2023-12-16T21:57:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.307+0000 7f4d5d8d0640 1 -- 172.21.15.28:0/504994835 wait complete. 2023-12-16T21:57:55.327 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-12-16T21:57:55.352 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-12-16T21:57:55.352 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-12-16T21:57:55.352 INFO:tasks.ceph_fuse:client.0 config is {} 2023-12-16T21:57:55.352 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:57:55.352 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:57:55.353 INFO:tasks.ceph_fuse:client.2 config is {} 2023-12-16T21:57:55.353 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:57:55.353 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-12-16T21:57:55.353 INFO:tasks.ceph_fuse:client.1 config is {} 2023-12-16T21:57:55.353 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:57:55.353 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-16T21:57:55.353 INFO:tasks.ceph_fuse:client.3 config is {} 2023-12-16T21:57:55.353 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:57:55.353 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-12-16T21:57:55.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:55.353 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2023-12-16T21:57:55.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:55.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link delete ceph-brx 2023-12-16T21:57:55.421 INFO:teuthology.orchestra.run.smithi122.stderr:Cannot find device "ceph-brx" 2023-12-16T21:57:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:57:55.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:55.422 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2023-12-16T21:57:55.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:55.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2023-12-16T21:57:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2023-12-16T21:57:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:57:55.489 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-12-16T21:57:55.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:57:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f093359e640 1 Processor -- start 2023-12-16T21:57:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f093359e640 1 -- start start 2023-12-16T21:57:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f093359e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09340cf300 0x7f093415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f093359e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093415f7e0 con 0x7f09340d18e0 2023-12-16T21:57:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f093359e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093415fac0 con 0x7f09340cef20 2023-12-16T21:57:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f093359e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09340b0880 con 0x7f09340cf300 2023-12-16T21:57:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f0931d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09340cf300 0x7f093415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f0932d9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f09340d18e0 0x7f09340b1060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51314/0 (socket says 172.21.15.28:51314) 2023-12-16T21:57:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f0932d9d640 1 -- 172.21.15.28:0/1118858053 learned_addr learned my addr 172.21.15.28:0/1118858053 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:57:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f0931d9b640 1 --2- 172.21.15.28:0/1118858053 >> v2:172.21.15.28:3300/0 conn(0x7f09340cf300 0x7f093415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46470/0 (socket says 172.21.15.28:46470) 2023-12-16T21:57:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f0931d9b640 1 -- 172.21.15.28:0/1118858053 >> v1:172.21.15.122:6789/0 conn(0x7f09340cef20 legacy=0x7f093415e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f0931d9b640 1 -- 172.21.15.28:0/1118858053 >> v1:172.21.15.28:6789/0 conn(0x7f09340d18e0 legacy=0x7f09340b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f0931d9b640 1 -- 172.21.15.28:0/1118858053 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09340b0b60 con 0x7f09340cf300 2023-12-16T21:57:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.663+0000 7f093159a640 1 -- 172.21.15.28:0/1118858053 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583152134 0 0) 0x7f093415f7e0 con 0x7f09340d18e0 2023-12-16T21:57:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f0931d9b640 1 --2- 172.21.15.28:0/1118858053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cf300 0x7f093415e820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f09280097c0 tx=0x7f092801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e541d59f755c1be7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093159a640 1 -- 172.21.15.28:0/1118858053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928004230 con 0x7f09340cf300 2023-12-16T21:57:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093159a640 1 -- 172.21.15.28:0/1118858053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09280a7050 con 0x7f09340cf300 2023-12-16T21:57:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093159a640 1 -- 172.21.15.28:0/1118858053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928003620 con 0x7f09340cf300 2023-12-16T21:57:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- 172.21.15.28:0/1118858053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cf300 msgr2=0x7f093415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 --2- 172.21.15.28:0/1118858053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cf300 0x7f093415e820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f09280097c0 tx=0x7f092801c440 comp rx=0 tx=0).stop 2023-12-16T21:57:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- 172.21.15.28:0/1118858053 shutdown_connections 2023-12-16T21:57:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 --2- 172.21.15.28:0/1118858053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cf300 0x7f093415e820 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- 172.21.15.28:0/1118858053 >> 172.21.15.28:0/1118858053 conn(0x7f09340d0dc0 msgr2=0x7f09340d11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- 172.21.15.28:0/1118858053 shutdown_connections 2023-12-16T21:57:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- 172.21.15.28:0/1118858053 wait complete. 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 Processor -- start 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- start start 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cef20 0x7f093414eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09340cf300 0x7f09341492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09340d18e0 0x7f09340b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0934149810 con 0x7f09340cef20 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093414dd00 con 0x7f09340cf300 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093414dfe0 con 0x7f09340d18e0 2023-12-16T21:57:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cef20 0x7f093414eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cef20 0x7f093414eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46482/0 (socket says 172.21.15.28:46482) 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 -- 172.21.15.28:0/249885319 learned_addr learned my addr 172.21.15.28:0/249885319 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f0931d9b640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09340cf300 0x7f09341492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f0932d9d640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09340d18e0 0x7f09340b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 -- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09340d18e0 msgr2=0x7f09340b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09340d18e0 0x7f09340b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 -- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09340cf300 msgr2=0x7f09341492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09340cf300 0x7f09341492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 -- 172.21.15.28:0/249885319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0928009420 con 0x7f09340cef20 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093259c640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cef20 0x7f093414eda0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f092400c450 tx=0x7f092400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f0922ffd640 1 -- 172.21.15.28:0/249885319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924012830 con 0x7f09340cef20 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f0922ffd640 1 -- 172.21.15.28:0/249885319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09240129b0 con 0x7f09340cef20 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093414e2c0 con 0x7f09340cef20 2023-12-16T21:57:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f0922ffd640 1 -- 172.21.15.28:0/249885319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924012d60 con 0x7f09340cef20 2023-12-16T21:57:55.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.667+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093414e880 con 0x7f09340cef20 2023-12-16T21:57:55.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.671+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093409ef30 con 0x7f09340cef20 2023-12-16T21:57:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.671+0000 7f0922ffd640 1 -- 172.21.15.28:0/249885319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f092401c460 con 0x7f09340cef20 2023-12-16T21:57:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.671+0000 7f0922ffd640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0914066470 0x7f0914068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.671+0000 7f0922ffd640 1 -- 172.21.15.28:0/249885319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0924083ca0 con 0x7f09340cef20 2023-12-16T21:57:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.671+0000 7f0931d9b640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0914066470 0x7f0914068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.671+0000 7f0931d9b640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0914066470 0x7f0914068930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f09280ad000 tx=0x7f0928008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.675+0000 7f0922ffd640 1 -- 172.21.15.28:0/249885319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f0924051120 con 0x7f09340cef20 2023-12-16T21:57:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.867+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f093404e930 con 0x7f09340cef20 2023-12-16T21:57:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.867+0000 7f0922ffd640 1 -- 172.21.15.28:0/249885319 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0924054740 con 0x7f09340cef20 2023-12-16T21:57:55.870 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0914066470 msgr2=0x7f0914068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0914066470 0x7f0914068930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f09280ad000 tx=0x7f0928008040 comp rx=0 tx=0).stop 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cef20 msgr2=0x7f093414eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cef20 0x7f093414eda0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f092400c450 tx=0x7f092400c910 comp rx=0 tx=0).stop 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 shutdown_connections 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0914066470 0x7f0914068930 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09340d18e0 0x7f09340b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09340cf300 0x7f09341492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 --2- 172.21.15.28:0/249885319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cef20 0x7f093414eda0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 >> 172.21.15.28:0/249885319 conn(0x7f09340d0dc0 msgr2=0x7f093406ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 shutdown_connections 2023-12-16T21:57:55.874 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:55.871+0000 7f093359e640 1 -- 172.21.15.28:0/249885319 wait complete. 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:57:55.890 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:57:55.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:55.891 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42451sec preferred_lft 42451sec 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:f4:d5:13:f2 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-16T21:57:55.898 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:55.899 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-16T21:57:55.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:55.899 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:55.899 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:55.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2023-12-16T21:57:55.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2023-12-16T21:57:55.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2023-12-16T21:57:55.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-16T21:57:55.899 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:55.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:55.977 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:57:56.028 INFO:teuthology.orchestra.run.smithi028.stdout:1 2023-12-16T21:57:56.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.029 DEBUG:teuthology.orchestra.run.smithi028:> route 2023-12-16T21:57:56.117 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2023-12-16T21:57:56.117 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:57:56.117 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-16T21:57:56.117 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-16T21:57:56.117 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-16T21:57:56.117 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:57:56.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.118 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:56.118 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:56.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:57:56.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:57:56.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:57:56.118 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:56.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.157 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2023-12-16T21:57:56.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.205 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2023-12-16T21:57:56.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.252 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:56.252 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:56.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:56.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-16T21:57:56.252 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:56.323 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-16T21:57:56.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.324 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:56.324 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:56.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-16T21:57:56.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-16T21:57:56.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-16T21:57:56.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-16T21:57:56.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-16T21:57:56.324 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:56.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.432 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:56.432 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:56.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2023-12-16T21:57:56.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2023-12-16T21:57:56.432 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:56.451 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:57:56.451 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:57:56.451 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:57:56.499 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:57:56.499 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:57:56.500 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:57:56.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2023-12-16T21:57:56.609 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:57:56.663 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2023-12-16T21:57:56.665 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2023-12-16T21:57:56.666 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2023-12-16T21:57:56.666 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2023-12-16T21:57:56.666 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2023-12-16T21:57:56.666 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2023-12-16T21:57:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:56.668 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:56.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.669 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2023-12-16T21:57:56.714 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:57:56.715 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:57:56.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2023-12-16T21:57:56.791 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:57:56.847 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2023-12-16T21:57:56.847 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2023-12-16T21:57:56.848 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:57:56.849 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:57:56.850 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:57:56.850 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2023-12-16T21:57:56.850 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:56.850 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:56.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.850 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2023-12-16T21:57:56.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2023-12-16T21:57:56.855+0000 7efe75f85580 -1 init, newargv = 0x5632a3ddfd30 newargc=15 2023-12-16T21:57:56.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[44367]: starting ceph client 2023-12-16T21:57:56.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[44367]: starting fuse 2023-12-16T21:57:56.867 INFO:teuthology.orchestra.run.smithi028.stdout:45 2023-12-16T21:57:56.867 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> import re 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> import os 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2023-12-16T21:57:56.868 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> return f 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2023-12-16T21:57:56.869 DEBUG:teuthology.orchestra.run.smithi028:> ' 2023-12-16T21:57:56.988 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.44367.asok 2023-12-16T21:57:56.991 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:57:56.999 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44367.asok 2023-12-16T21:57:56.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:56.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44367.asok status 2023-12-16T21:57:57.164 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-16T21:57:57.164 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2023-12-16T21:57:57.164 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "44367", 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4635, 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2023-12-16T21:57:57.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4635 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1641427043 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1641427043 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:57:57.166 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.4635 192.168.0.1:0/1641427043", 2023-12-16T21:57:57.167 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/1641427043", 2023-12-16T21:57:57.167 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2023-12-16T21:57:57.167 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 8, 2023-12-16T21:57:57.167 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 23, 2023-12-16T21:57:57.167 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:57:57.167 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2023-12-16T21:57:57.167 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2023-12-16T21:57:57.167 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-16T21:57:57.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:57:57.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b6fc4640 1 Processor -- start 2023-12-16T21:57:57.310 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b6fc4640 1 -- start start 2023-12-16T21:57:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b6fc4640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35b0144840 0x7f35b006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b6fc4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0072180 0x7f35b006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b6fc4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b00710e0 con 0x7f35b0070460 2023-12-16T21:57:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b6fc4640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b015f010 con 0x7f35b0144840 2023-12-16T21:57:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b6fc4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b00713c0 con 0x7f35b0072180 2023-12-16T21:57:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b4d60640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35b0144840 0x7f35b006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b4d60640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35b0144840 0x7f35b006c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46490/0 (socket says 172.21.15.28:46490) 2023-12-16T21:57:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35b4d60640 1 -- 172.21.15.28:0/1547972318 learned_addr learned my addr 172.21.15.28:0/1547972318 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.307+0000 7f35a7fff640 1 --2- 172.21.15.28:0/1547972318 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0072180 0x7f35b006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b4d60640 1 -- 172.21.15.28:0/1547972318 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0072180 msgr2=0x7f35b006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:57:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b4d60640 1 --2- 172.21.15.28:0/1547972318 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0072180 0x7f35b006c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b4d60640 1 -- 172.21.15.28:0/1547972318 >> v1:172.21.15.122:6789/0 conn(0x7f35b0070460 legacy=0x7f35b00708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b4d60640 1 -- 172.21.15.28:0/1547972318 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b00716a0 con 0x7f35b0144840 2023-12-16T21:57:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b4d60640 1 --2- 172.21.15.28:0/1547972318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b006c2c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009870 tx=0x7f35ac01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb34f34c35d6263 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35a77fe640 1 -- 172.21.15.28:0/1547972318 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0aa670 con 0x7f35b0144840 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35a77fe640 1 -- 172.21.15.28:0/1547972318 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac0aa7d0 con 0x7f35b0144840 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35a77fe640 1 -- 172.21.15.28:0/1547972318 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0aab60 con 0x7f35b0144840 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b6fc4640 1 -- 172.21.15.28:0/1547972318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 msgr2=0x7f35b006c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/1547972318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b006c2c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009870 tx=0x7f35ac01c510 comp rx=0 tx=0).stop 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b6fc4640 1 -- 172.21.15.28:0/1547972318 shutdown_connections 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/1547972318 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0072180 0x7f35b006c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/1547972318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b006c2c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b6fc4640 1 -- 172.21.15.28:0/1547972318 >> 172.21.15.28:0/1547972318 conn(0x7f35b006a530 msgr2=0x7f35b006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b6fc4640 1 -- 172.21.15.28:0/1547972318 shutdown_connections 2023-12-16T21:57:57.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.311+0000 7f35b6fc4640 1 -- 172.21.15.28:0/1547972318 wait complete. 2023-12-16T21:57:57.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 Processor -- start 2023-12-16T21:57:57.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 -- start start 2023-12-16T21:57:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0070460 0x7f35b0140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35b0072180 0x7f35b0158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b01597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b01440f0 con 0x7f35b0144840 2023-12-16T21:57:57.321 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b0160840 con 0x7f35b0072180 2023-12-16T21:57:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b0159350 con 0x7f35b0070460 2023-12-16T21:57:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b01597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b01597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46504/0 (socket says 172.21.15.28:46504) 2023-12-16T21:57:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 -- 172.21.15.28:0/3393393079 learned_addr learned my addr 172.21.15.28:0/3393393079 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:57:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35a7fff640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35b0072180 0x7f35b0158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b4d60640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0070460 0x7f35b0140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 -- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0070460 msgr2=0x7f35b0140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0070460 0x7f35b0140bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 -- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35b0072180 msgr2=0x7f35b0158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35b0072180 0x7f35b0158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 -- 172.21.15.28:0/3393393079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0094d0 con 0x7f35b0144840 2023-12-16T21:57:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b5561640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b01597e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35a800c7a0 tx=0x7f35a800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35a57fa640 1 -- 172.21.15.28:0/3393393079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8003b80 con 0x7f35b0144840 2023-12-16T21:57:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35a57fa640 1 -- 172.21.15.28:0/3393393079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8003ce0 con 0x7f35b0144840 2023-12-16T21:57:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b01410f0 con 0x7f35b0144840 2023-12-16T21:57:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35a57fa640 1 -- 172.21.15.28:0/3393393079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a800aa30 con 0x7f35b0144840 2023-12-16T21:57:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b0158d00 con 0x7f35b0144840 2023-12-16T21:57:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b01594e0 con 0x7f35b0144840 2023-12-16T21:57:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.315+0000 7f35a57fa640 1 -- 172.21.15.28:0/3393393079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f35a800ab90 con 0x7f35b0144840 2023-12-16T21:57:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.319+0000 7f35a57fa640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f359c0664a0 0x7f359c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:57:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.319+0000 7f35a57fa640 1 -- 172.21.15.28:0/3393393079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35a8083b20 con 0x7f35b0144840 2023-12-16T21:57:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.319+0000 7f35b4d60640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f359c0664a0 0x7f359c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:57:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.319+0000 7f35b4d60640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f359c0664a0 0x7f359c068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0ae000 tx=0x7f35ac0b5040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:57:57.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.323+0000 7f35a57fa640 1 -- 172.21.15.28:0/3393393079 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f35a8051050 con 0x7f35b0144840 2023-12-16T21:57:57.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f35b0069440 con 0x7f35b0144840 2023-12-16T21:57:57.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35a57fa640 1 -- 172.21.15.28:0/3393393079 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f35a8054670 con 0x7f35b0144840 2023-12-16T21:57:57.517 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:57:57.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f359c0664a0 msgr2=0x7f359c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:57.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f359c0664a0 0x7f359c068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0ae000 tx=0x7f35ac0b5040 comp rx=0 tx=0).stop 2023-12-16T21:57:57.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 msgr2=0x7f35b01597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:57:57.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b01597e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35a800c7a0 tx=0x7f35a800cc60 comp rx=0 tx=0).stop 2023-12-16T21:57:57.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 shutdown_connections 2023-12-16T21:57:57.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f359c0664a0 0x7f359c068960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b0144840 0x7f35b01597e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35b0072180 0x7f35b0158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 --2- 172.21.15.28:0/3393393079 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f35b0070460 0x7f35b0140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:57:57.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 >> 172.21.15.28:0/3393393079 conn(0x7f35b006a530 msgr2=0x7f35b015d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:57:57.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 shutdown_connections 2023-12-16T21:57:57.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:57.515+0000 7f35b6fc4640 1 -- 172.21.15.28:0/3393393079 wait complete. 2023-12-16T21:57:57.537 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:57:57.538 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:57:57.538 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2023-12-16T21:57:57.538 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-12-16T21:57:57.538 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-12-16T21:57:57.538 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:57:57.538 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:57:57.538 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:57:57.538 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-12-16T21:57:57.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.538 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:57:57.545 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42449sec preferred_lft 42449sec 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:f4:d5:13:f2 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.546 INFO:teuthology.orchestra.run.smithi028.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link tentative 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.547 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:57:57.598 INFO:teuthology.orchestra.run.smithi028.stdout:1 2023-12-16T21:57:57.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.600 DEBUG:teuthology.orchestra.run.smithi028:> route 2023-12-16T21:57:57.653 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2023-12-16T21:57:57.653 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:57:57.653 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-16T21:57:57.653 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-16T21:57:57.653 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-16T21:57:57.653 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:57:57.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.654 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:57.654 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:57.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:57:57.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:57:57.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:57:57.654 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:57.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.729 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2023-12-16T21:57:57.780 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:57:57.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.780 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2023-12-16T21:57:57.828 INFO:teuthology.orchestra.run.smithi028.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-12-16T21:57:57.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.829 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:57.829 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:57.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:57:57.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-12-16T21:57:57.829 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:57.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-16T21:57:57.952 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:57:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:57:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:57:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2023-12-16T21:57:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.952 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:57:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 86:fc:e3:76:b7:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-16T21:57:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-16T21:57:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::84fc:e3ff:fe76:b78d/64 scope link 2023-12-16T21:57:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:57.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:57.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-16T21:57:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:57:58.013 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:57:58.013 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:57:58.013 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:58.013 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2023-12-16T21:57:58.013 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:58.013 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:57:58.014 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 86:fc:e3:76:b7:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-16T21:57:58.014 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-16T21:57:58.014 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:58.014 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::84fc:e3ff:fe76:b78d/64 scope link 2023-12-16T21:57:58.014 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:57:58.015 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-12-16T21:57:58.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:58.015 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:58.015 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:58.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-12-16T21:57:58.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-16T21:57:58.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-12-16T21:57:58.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-12-16T21:57:58.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-12-16T21:57:58.016 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:58.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:58.118 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:57:58.118 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:57:58.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.1 up 2023-12-16T21:57:58.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.1 master ceph-brx 2023-12-16T21:57:58.118 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:57:58.139 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-12-16T21:57:58.139 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:57:58.140 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-12-16T21:57:58.192 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-12-16T21:57:58.192 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:57:58.192 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-12-16T21:57:58.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2023-12-16T21:57:58.296 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2023-12-16T21:57:58.346 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2023-12-16T21:57:58.347 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:57:58.348 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:57:58.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:58.349 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2023-12-16T21:57:58.396 INFO:teuthology.orchestra.run.smithi028.stdout:45 2023-12-16T21:57:58.396 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [45] 2023-12-16T21:57:58.396 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-12-16T21:57:58.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2023-12-16T21:57:58.456 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:57:58.504 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2023-12-16T21:57:58.504 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2023-12-16T21:57:58.504 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2023-12-16T21:57:58.505 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2023-12-16T21:57:58.506 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:58.507 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:58.507 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-16T21:57:58.507 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:57:58.507 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:57:58.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:58.508 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2023-12-16T21:57:58.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi028.stderr:2023-12-16T21:57:58.527+0000 7f9490726580 -1 init, newargv = 0x55bfaf158780 newargc=15 2023-12-16T21:57:58.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi028.stderr:ceph-fuse[44502]: starting ceph client 2023-12-16T21:57:58.532 INFO:teuthology.orchestra.run.smithi028.stdout:45 2023-12-16T21:57:58.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi028.stderr:ceph-fuse[44502]: starting fuse 2023-12-16T21:57:59.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2023-12-16T21:57:59.653 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2023-12-16T21:57:59.702 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2023-12-16T21:57:59.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:57:59.704 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.2 2023-12-16T21:57:59.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:59.705 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2023-12-16T21:57:59.752 INFO:teuthology.orchestra.run.smithi028.stdout:45 2023-12-16T21:57:59.752 INFO:teuthology.orchestra.run.smithi028.stdout:46 2023-12-16T21:57:59.753 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45, 46] 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> import re 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> import os 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:57:59.753 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> return f 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.2")) 2023-12-16T21:57:59.754 DEBUG:teuthology.orchestra.run.smithi028:> ' 2023-12-16T21:57:59.879 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.2.44502.asok 2023-12-16T21:57:59.883 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:57:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:57:59.891 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.44502.asok 2023-12-16T21:57:59.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:57:59.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.44502.asok status 2023-12-16T21:58:00.055 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-16T21:58:00.055 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2023-12-16T21:58:00.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:58:00.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:58:00.055 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "2", 2023-12-16T21:58:00.055 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2023-12-16T21:58:00.055 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "44502", 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4647, 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4647 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.2:0", 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3289521487 2023-12-16T21:58:00.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.2:0", 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3289521487 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.4647 192.168.0.2:0/3289521487", 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.2:0/3289521487", 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 8, 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 23, 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2023-12-16T21:58:00.057 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2023-12-16T21:58:00.058 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-16T21:58:00.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:58:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe9a23640 1 Processor -- start 2023-12-16T21:58:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe9a23640 1 -- start start 2023-12-16T21:58:00.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe9a23640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fe40b2ed0 0x7f5fe4071430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe9a23640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5fe4071970 0x7f5fe415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe9a23640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe40af880 con 0x7f5fe41579e0 2023-12-16T21:58:00.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe9a23640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe4071e00 con 0x7f5fe4071970 2023-12-16T21:58:00.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe9a23640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe415a8c0 con 0x7f5fe40b2ed0 2023-12-16T21:58:00.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe2ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5fe41579e0 0x7f5fe40b4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59022/0 (socket says 172.21.15.28:59022) 2023-12-16T21:58:00.198 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe2ffd640 1 -- 172.21.15.28:0/3415386701 learned_addr learned my addr 172.21.15.28:0/3415386701 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe37fe640 1 --2- 172.21.15.28:0/3415386701 >> v2:172.21.15.122:3300/0 conn(0x7f5fe4071970 0x7f5fe415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183146666 0 0) 0x7f5fe40af880 con 0x7f5fe41579e0 2023-12-16T21:58:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe27fc640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fe40b2ed0 0x7f5fe4071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd0003530 con 0x7f5fe41579e0 2023-12-16T21:58:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2362010973 0 0) 0x7f5fd0003530 con 0x7f5fe41579e0 2023-12-16T21:58:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fe40b2ed0 msgr2=0x7f5fe4071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fe40b2ed0 0x7f5fe4071430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 >> v2:172.21.15.122:3300/0 conn(0x7f5fe4071970 msgr2=0x7f5fe415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 --2- 172.21.15.28:0/3415386701 >> v2:172.21.15.122:3300/0 conn(0x7f5fe4071970 0x7f5fe415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe415aba0 con 0x7f5fe41579e0 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f5fd4001680 con 0x7f5fe41579e0 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 >> v1:172.21.15.28:6789/0 conn(0x7f5fe41579e0 legacy=0x7f5fe40b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fe40b2ed0 0x7f5fd0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd0003f00 0x7f5fd00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd0004d10 0x7f5fd0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4071e00 con 0x7f5fd0004d10 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe415a8c0 con 0x7f5fd0003f00 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd000b9e0 con 0x7f5fe40b2ed0 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe37fe640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd0004d10 0x7f5fd0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe2ffd640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fe40b2ed0 0x7f5fd0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe27fc640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd0003f00 0x7f5fd00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe37fe640 1 -- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fe40b2ed0 msgr2=0x7f5fd0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe37fe640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fe40b2ed0 0x7f5fd0003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe37fe640 1 -- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd0003f00 msgr2=0x7f5fd00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe37fe640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd0003f00 0x7f5fd00047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.195+0000 7f5fe37fe640 1 -- 172.21.15.28:0/3415386701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd0003530 con 0x7f5fd0004d10 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe37fe640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd0004d10 0x7f5fd0005190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009b10 tx=0x7f5fdc01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c7013ae12b5a4c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0a5680 con 0x7f5fd0004d10 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc0a57e0 con 0x7f5fd0004d10 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe1ffb640 1 -- 172.21.15.28:0/3415386701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0a5b90 con 0x7f5fd0004d10 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe9a23640 1 -- 172.21.15.28:0/3415386701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd0004d10 msgr2=0x7f5fd0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe9a23640 1 --2- 172.21.15.28:0/3415386701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fd0004d10 0x7f5fd0005190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009b10 tx=0x7f5fdc01ca90 comp rx=0 tx=0).stop 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe37fe640 1 -- 172.21.15.28:0/3415386701 reap_dead start 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe9a23640 1 -- 172.21.15.28:0/3415386701 shutdown_connections 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe9a23640 1 -- 172.21.15.28:0/3415386701 >> 172.21.15.28:0/3415386701 conn(0x7f5fe40b1bd0 msgr2=0x7f5fe40b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe9a23640 1 -- 172.21.15.28:0/3415386701 shutdown_connections 2023-12-16T21:58:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe9a23640 1 -- 172.21.15.28:0/3415386701 wait complete. 2023-12-16T21:58:00.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe9a23640 1 Processor -- start 2023-12-16T21:58:00.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.199+0000 7f5fe9a23640 1 -- start start 2023-12-16T21:58:00.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fd0004d10 0x7f5fe414cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe414c200 0x7f5fe41f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fe414ac50 0x7f5fe414b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe414d190 con 0x7f5fe414ac50 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe414b610 con 0x7f5fe414c200 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe41f2870 con 0x7f5fd0004d10 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fe414ac50 0x7f5fe414b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fe414ac50 0x7f5fe414b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38292/0 (socket says 172.21.15.28:38292) 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 -- 172.21.15.28:0/731216815 learned_addr learned my addr 172.21.15.28:0/731216815 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe2ffd640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fd0004d10 0x7f5fe414cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe27fc640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe414c200 0x7f5fe41f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 -- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fd0004d10 msgr2=0x7f5fe414cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fd0004d10 0x7f5fe414cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 -- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe414c200 msgr2=0x7f5fe41f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe414c200 0x7f5fe41f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 -- 172.21.15.28:0/731216815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe414d3a0 con 0x7f5fe414ac50 2023-12-16T21:58:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe37fe640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fe414ac50 0x7f5fe414b0d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0b0040 tx=0x7f5fd0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fdbfff640 1 -- 172.21.15.28:0/731216815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0ba070 con 0x7f5fe414ac50 2023-12-16T21:58:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fdbfff640 1 -- 172.21.15.28:0/731216815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc004020 con 0x7f5fe414ac50 2023-12-16T21:58:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe4149bb0 con 0x7f5fe414ac50 2023-12-16T21:58:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fdbfff640 1 -- 172.21.15.28:0/731216815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0a5680 con 0x7f5fe414ac50 2023-12-16T21:58:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe414a170 con 0x7f5fe414ac50 2023-12-16T21:58:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe40674d0 con 0x7f5fe414ac50 2023-12-16T21:58:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fdbfff640 1 -- 172.21.15.28:0/731216815 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f5fdc0b4070 con 0x7f5fe414ac50 2023-12-16T21:58:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fdbfff640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5fc4066470 0x7f5fc4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.203+0000 7f5fdbfff640 1 -- 172.21.15.28:0/731216815 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fdc11da30 con 0x7f5fe414ac50 2023-12-16T21:58:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.207+0000 7f5fe2ffd640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5fc4066470 0x7f5fc4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.207+0000 7f5fe2ffd640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5fc4066470 0x7f5fc4068930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40d0e80 tx=0x7f5fd4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:00.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.211+0000 7f5fdbfff640 1 -- 172.21.15.28:0/731216815 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f5fdc0eb080 con 0x7f5fe414ac50 2023-12-16T21:58:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.399+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5fe40674d0 con 0x7f5fe414ac50 2023-12-16T21:58:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fdbfff640 1 -- 172.21.15.28:0/731216815 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5fdc0043b0 con 0x7f5fe414ac50 2023-12-16T21:58:00.403 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5fc4066470 msgr2=0x7f5fc4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5fc4066470 0x7f5fc4068930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40d0e80 tx=0x7f5fd4009040 comp rx=0 tx=0).stop 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fe414ac50 msgr2=0x7f5fe414b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fe414ac50 0x7f5fe414b0d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0b0040 tx=0x7f5fd0005800 comp rx=0 tx=0).stop 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 shutdown_connections 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5fc4066470 0x7f5fc4068930 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fe414ac50 0x7f5fe414b0d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fe414c200 0x7f5fe41f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 --2- 172.21.15.28:0/731216815 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5fd0004d10 0x7f5fe414cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 >> 172.21.15.28:0/731216815 conn(0x7f5fe40b1bd0 msgr2=0x7f5fe40af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 shutdown_connections 2023-12-16T21:58:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:00.403+0000 7f5fe9a23640 1 -- 172.21.15.28:0/731216815 wait complete. 2023-12-16T21:58:00.422 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:58:00.422 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:58:00.422 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2023-12-16T21:58:00.423 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-16T21:58:00.423 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:00.423 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:58:00.423 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:58:00.423 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:58:00.423 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-16T21:58:00.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.423 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2023-12-16T21:58:00.430 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:58:00.430 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:58:00.430 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:58:00.430 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42454sec preferred_lft 42454sec 2023-12-16T21:58:00.431 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2023-12-16T21:58:00.432 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:00.432 INFO:teuthology.orchestra.run.smithi122.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-16T21:58:00.432 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 02:42:be:b5:cb:42 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:00.432 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-16T21:58:00.432 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:00.432 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-16T21:58:00.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.432 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:00.432 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:00.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add name ceph-brx type bridge 2023-12-16T21:58:00.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip addr flush dev ceph-brx 2023-12-16T21:58:00.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set ceph-brx up 2023-12-16T21:58:00.433 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-16T21:58:00.433 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:00.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.514 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:58:00.565 INFO:teuthology.orchestra.run.smithi122.stdout:1 2023-12-16T21:58:00.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.567 DEBUG:teuthology.orchestra.run.smithi122:> route 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-16T21:58:00.631 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:58:00.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.632 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:00.632 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:00.632 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-16T21:58:00.632 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-16T21:58:00.632 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-16T21:58:00.632 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:00.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.704 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2023-12-16T21:58:00.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.753 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list-id 2023-12-16T21:58:00.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.800 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:00.800 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:00.800 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:00.801 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-16T21:58:00.801 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:00.870 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-16T21:58:00.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.871 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:00.871 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:00.871 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-16T21:58:00.871 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-16T21:58:00.871 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-16T21:58:00.871 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-16T21:58:00.871 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-16T21:58:00.871 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:00.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:00.976 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:00.977 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:00.977 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2023-12-16T21:58:00.977 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set dev brx.0 master ceph-brx 2023-12-16T21:58:00.977 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:00.996 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-12-16T21:58:00.996 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:58:00.996 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:01.048 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-16T21:58:01.048 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:58:01.048 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:01.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2023-12-16T21:58:01.154 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:01.200 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2023-12-16T21:58:01.200 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2023-12-16T21:58:01.200 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2023-12-16T21:58:01.200 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2023-12-16T21:58:01.200 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/run/lock 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/fs/nfsd 2023-12-16T21:58:01.201 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/run/rpc_pipefs 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:01.202 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:01.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:01.203 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2023-12-16T21:58:01.252 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:58:01.252 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-12-16T21:58:01.297 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2023-12-16T21:58:01.311 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:01.361 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2023-12-16T21:58:01.361 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2023-12-16T21:58:01.361 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2023-12-16T21:58:01.361 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2023-12-16T21:58:01.361 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2023-12-16T21:58:01.361 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/run/lock 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/fs/nfsd 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2023-12-16T21:58:01.362 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/run/rpc_pipefs 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:01.363 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:01.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:01.364 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2023-12-16T21:58:01.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:2023-12-16T21:58:01.383+0000 7f1841c2e580 -1 init, newargv = 0x55a265940410 newargc=15 2023-12-16T21:58:01.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:ceph-fuse[44346]: starting ceph client 2023-12-16T21:58:01.392 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:ceph-fuse[44346]: starting fuse 2023-12-16T21:58:02.492 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2023-12-16T21:58:02.509 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/run/lock 2023-12-16T21:58:02.563 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/fs/nfsd 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/run/rpc_pipefs 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2023-12-16T21:58:02.564 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2023-12-16T21:58:02.565 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:58:02.565 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:58:02.565 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:58:02.565 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:58:02.565 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2023-12-16T21:58:02.565 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:02.565 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:02.565 INFO:teuthology.orchestra.run.smithi122.stdout:/home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:02.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:02.566 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2023-12-16T21:58:02.616 INFO:teuthology.orchestra.run.smithi122.stdout:45 2023-12-16T21:58:02.617 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> import re 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> import os 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-12-16T21:58:02.617 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> return f 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> 2023-12-16T21:58:02.618 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.1")) 2023-12-16T21:58:02.619 DEBUG:teuthology.orchestra.run.smithi122:> ' 2023-12-16T21:58:02.739 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.1.44346.asok 2023-12-16T21:58:02.743 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:58:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:58:02.750 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44346.asok 2023-12-16T21:58:02.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:02.750 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44346.asok status 2023-12-16T21:58:02.905 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "1", 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "44346", 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 4442, 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2023-12-16T21:58:02.906 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2023-12-16T21:58:02.908 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2023-12-16T21:58:02.908 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 4442 2023-12-16T21:58:02.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-12-16T21:58:02.908 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2023-12-16T21:58:02.908 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2023-12-16T21:58:02.908 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 657558762 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 657558762 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.4442 192.168.0.1:0/657558762", 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/657558762", 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 8, 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 23, 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:58:02.909 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2023-12-16T21:58:02.910 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2023-12-16T21:58:02.910 INFO:teuthology.orchestra.run.smithi122.stdout:} 2023-12-16T21:58:02.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:58:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.051+0000 7fe627118640 1 Processor -- start 2023-12-16T21:58:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.051+0000 7fe627118640 1 -- start start 2023-12-16T21:58:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe627118640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe62006c9d0 0x7fe620070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe627118640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe620072180 0x7fe6200709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe627118640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6200710e0 con 0x7fe620144840 2023-12-16T21:58:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe627118640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62015f010 con 0x7fe62006c9d0 2023-12-16T21:58:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe627118640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe620070ee0 con 0x7fe620072180 2023-12-16T21:58:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe624eb4640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe620144840 0x7fe62006c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59030/0 (socket says 172.21.15.28:59030) 2023-12-16T21:58:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe624eb4640 1 -- 172.21.15.28:0/2496956468 learned_addr learned my addr 172.21.15.28:0/2496956468 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe6256b5640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe620072180 0x7fe6200709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61ffff640 1 --2- 172.21.15.28:0/2496956468 >> v2:172.21.15.122:3300/0 conn(0x7fe62006c9d0 0x7fe620070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263770016 0 0) 0x7fe6200710e0 con 0x7fe620144840 2023-12-16T21:58:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe608003530 con 0x7fe620144840 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 554106072 0 0) 0x7fe608003530 con 0x7fe620144840 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe620072180 msgr2=0x7fe6200709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe620072180 0x7fe6200709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 >> v2:172.21.15.122:3300/0 conn(0x7fe62006c9d0 msgr2=0x7fe620070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 --2- 172.21.15.28:0/2496956468 >> v2:172.21.15.122:3300/0 conn(0x7fe62006c9d0 0x7fe620070460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6200713c0 con 0x7fe620144840 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fe610001680 con 0x7fe620144840 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 >> v1:172.21.15.28:6789/0 conn(0x7fe620144840 legacy=0x7fe62006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe620072180 0x7fe608003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe608003f00 0x7fe6080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608004d10 0x7fe608005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe620070ee0 con 0x7fe608003f00 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60800b990 con 0x7fe608004d10 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60800bad0 con 0x7fe620072180 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61ffff640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe608003f00 0x7fe6080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe6256b5640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608004d10 0x7fe608005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe624eb4640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe620072180 0x7fe608003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61ffff640 1 -- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe620072180 msgr2=0x7fe608003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61ffff640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe620072180 0x7fe608003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61ffff640 1 -- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608004d10 msgr2=0x7fe608005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61ffff640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608004d10 0x7fe608005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe61ffff640 1 -- 172.21.15.28:0/2496956468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608003530 con 0x7fe608003f00 2023-12-16T21:58:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.055+0000 7fe6256b5640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608004d10 0x7fe608005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe61ffff640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe608003f00 0x7fe6080047d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe614009ab0 tx=0x7fe61401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2791df27155c2229 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6140a5660 con 0x7fe608003f00 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6140a57c0 con 0x7fe608003f00 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe61f7fe640 1 -- 172.21.15.28:0/2496956468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6140a5b70 con 0x7fe608003f00 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- 172.21.15.28:0/2496956468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe608003f00 msgr2=0x7fe6080047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 --2- 172.21.15.28:0/2496956468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe608003f00 0x7fe6080047d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe614009ab0 tx=0x7fe61401ca10 comp rx=0 tx=0).stop 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe6256b5640 1 -- 172.21.15.28:0/2496956468 reap_dead start 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- 172.21.15.28:0/2496956468 shutdown_connections 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- 172.21.15.28:0/2496956468 >> 172.21.15.28:0/2496956468 conn(0x7fe62006a530 msgr2=0x7fe62006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- 172.21.15.28:0/2496956468 shutdown_connections 2023-12-16T21:58:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- 172.21.15.28:0/2496956468 wait complete. 2023-12-16T21:58:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 Processor -- start 2023-12-16T21:58:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- start start 2023-12-16T21:58:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62015a7b0 0x7fe6200c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe62015ab90 0x7fe62013c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6200c5950 0x7fe62013cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62013a410 con 0x7fe62015a7b0 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62013d670 con 0x7fe62015ab90 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe627118640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62013d910 con 0x7fe6200c5950 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe624eb4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62015a7b0 0x7fe6200c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe624eb4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62015a7b0 0x7fe6200c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38316/0 (socket says 172.21.15.28:38316) 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe624eb4640 1 -- 172.21.15.28:0/2145124711 learned_addr learned my addr 172.21.15.28:0/2145124711 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe61ffff640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe62015ab90 0x7fe62013c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe6256b5640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6200c5950 0x7fe62013cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe624eb4640 1 -- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6200c5950 msgr2=0x7fe62013cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe624eb4640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6200c5950 0x7fe62013cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe624eb4640 1 -- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe62015ab90 msgr2=0x7fe62013c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe624eb4640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe62015ab90 0x7fe62013c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.059+0000 7fe624eb4640 1 -- 172.21.15.28:0/2145124711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62013dbf0 con 0x7fe62015a7b0 2023-12-16T21:58:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe624eb4640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62015a7b0 0x7fe6200c5410 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe610010bd0 tx=0x7fe61000e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe61d7fa640 1 -- 172.21.15.28:0/2145124711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000ece0 con 0x7fe62015a7b0 2023-12-16T21:58:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe61d7fa640 1 -- 172.21.15.28:0/2145124711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61000ee40 con 0x7fe62015a7b0 2023-12-16T21:58:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe61d7fa640 1 -- 172.21.15.28:0/2145124711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6100145a0 con 0x7fe62015a7b0 2023-12-16T21:58:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62013ded0 con 0x7fe62015a7b0 2023-12-16T21:58:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62013e6d0 con 0x7fe62015a7b0 2023-12-16T21:58:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe620069440 con 0x7fe62015a7b0 2023-12-16T21:58:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe61d7fa640 1 -- 172.21.15.28:0/2145124711 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe610014910 con 0x7fe62015a7b0 2023-12-16T21:58:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe61d7fa640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe60c0664a0 0x7fe60c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe61d7fa640 1 -- 172.21.15.28:0/2145124711 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe610086bc0 con 0x7fe62015a7b0 2023-12-16T21:58:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe61ffff640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe60c0664a0 0x7fe60c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.063+0000 7fe61ffff640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe60c0664a0 0x7fe60c068960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe6140af000 tx=0x7fe6140046d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.067+0000 7fe61d7fa640 1 -- 172.21.15.28:0/2145124711 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe610054180 con 0x7fe62015a7b0 2023-12-16T21:58:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.259+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe62006d170 con 0x7fe62015a7b0 2023-12-16T21:58:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe61d7fa640 1 -- 172.21.15.28:0/2145124711 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe61001b070 con 0x7fe62015a7b0 2023-12-16T21:58:03.263 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:58:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe60c0664a0 msgr2=0x7fe60c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe60c0664a0 0x7fe60c068960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe6140af000 tx=0x7fe6140046d0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62015a7b0 msgr2=0x7fe6200c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62015a7b0 0x7fe6200c5410 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe610010bd0 tx=0x7fe61000e4d0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 shutdown_connections 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe60c0664a0 0x7fe60c068960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6200c5950 0x7fe62013cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe62015ab90 0x7fe62013c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 --2- 172.21.15.28:0/2145124711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe62015a7b0 0x7fe6200c5410 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 >> 172.21.15.28:0/2145124711 conn(0x7fe62006a530 msgr2=0x7fe620070670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 shutdown_connections 2023-12-16T21:58:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:03.263+0000 7fe627118640 1 -- 172.21.15.28:0/2145124711 wait complete. 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:58:03.283 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-12-16T21:58:03.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.283 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2023-12-16T21:58:03.290 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:58:03.290 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:58:03.290 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-12-16T21:58:03.291 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 42451sec preferred_lft 42451sec 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 02:42:be:b5:cb:42 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.292 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-12-16T21:58:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.293 INFO:teuthology.orchestra.run.smithi122.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:58:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-12-16T21:58:03.293 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.294 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:58:03.344 INFO:teuthology.orchestra.run.smithi122.stdout:1 2023-12-16T21:58:03.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.346 DEBUG:teuthology.orchestra.run.smithi122:> route 2023-12-16T21:58:03.395 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2023-12-16T21:58:03.395 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:58:03.395 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-12-16T21:58:03.396 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-16T21:58:03.396 INFO:teuthology.orchestra.run.smithi122.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-16T21:58:03.396 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-16T21:58:03.396 INFO:teuthology.orchestra.run.smithi122.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-16T21:58:03.396 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-16T21:58:03.396 INFO:teuthology.orchestra.run.smithi122.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-12-16T21:58:03.397 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:58:03.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.398 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:03.398 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:03.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-16T21:58:03.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-16T21:58:03.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-16T21:58:03.399 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:03.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.473 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2023-12-16T21:58:03.521 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-16T21:58:03.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.522 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list-id 2023-12-16T21:58:03.568 INFO:teuthology.orchestra.run.smithi122.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-12-16T21:58:03.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.569 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:03.569 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:03.569 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:58:03.569 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-12-16T21:58:03.569 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:03.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.642 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-16T21:58:03.695 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether be:80:a6:06:ae:54 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::bc80:a6ff:fe06:ae54/64 scope link 2023-12-16T21:58:03.696 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.698 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether be:80:a6:06:ae:54 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-16T21:58:03.753 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.754 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::bc80:a6ff:fe06:ae54/64 scope link 2023-12-16T21:58:03.754 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:03.755 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-12-16T21:58:03.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.755 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:03.755 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:03.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-12-16T21:58:03.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-16T21:58:03.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-12-16T21:58:03.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-12-16T21:58:03.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-12-16T21:58:03.755 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:03.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:03.852 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec bash -c ' 2023-12-16T21:58:03.852 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2023-12-16T21:58:03.852 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.1 up 2023-12-16T21:58:03.853 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set dev brx.1 master ceph-brx 2023-12-16T21:58:03.853 DEBUG:teuthology.orchestra.run.smithi122:> ') 2023-12-16T21:58:03.873 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-12-16T21:58:03.873 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:58:03.873 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-12-16T21:58:03.920 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-12-16T21:58:03.920 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:58:03.921 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-12-16T21:58:03.972 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2023-12-16T21:58:04.034 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:04.082 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2023-12-16T21:58:04.082 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2023-12-16T21:58:04.082 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/run/lock 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2023-12-16T21:58:04.083 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/fs/nfsd 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/run/rpc_pipefs 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:04.084 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:58:04.085 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:58:04.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:04.086 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2023-12-16T21:58:04.132 INFO:teuthology.orchestra.run.smithi122.stdout:45 2023-12-16T21:58:04.132 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [45] 2023-12-16T21:58:04.132 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-12-16T21:58:04.177 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2023-12-16T21:58:04.190 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:04.237 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2023-12-16T21:58:04.237 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2023-12-16T21:58:04.237 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2023-12-16T21:58:04.237 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2023-12-16T21:58:04.237 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2023-12-16T21:58:04.237 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2023-12-16T21:58:04.237 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/run/lock 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/fs/nfsd 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:04.238 INFO:teuthology.orchestra.run.smithi122.stdout:/run/rpc_pipefs 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:58:04.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:04.240 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2023-12-16T21:58:04.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi122.stderr:2023-12-16T21:58:04.250+0000 7fbf3d060580 -1 init, newargv = 0x55ef313d06f0 newargc=15 2023-12-16T21:58:04.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi122.stderr:ceph-fuse[44458]: starting ceph client 2023-12-16T21:58:04.258 INFO:teuthology.orchestra.run.smithi122.stdout:45 2023-12-16T21:58:04.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi122.stderr:ceph-fuse[44458]: starting fuse 2023-12-16T21:58:05.359 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2023-12-16T21:58:05.375 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:05.421 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2023-12-16T21:58:05.421 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2023-12-16T21:58:05.421 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2023-12-16T21:58:05.421 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/run/lock 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2023-12-16T21:58:05.422 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/fs/nfsd 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/run/rpc_pipefs 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:58:05.423 INFO:teuthology.orchestra.run.smithi122.stdout:/home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:05.424 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:58:05.424 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:58:05.424 INFO:teuthology.orchestra.run.smithi122.stdout:/home/ubuntu/cephtest/mnt.3 2023-12-16T21:58:05.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:05.427 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2023-12-16T21:58:05.472 INFO:teuthology.orchestra.run.smithi122.stdout:45 2023-12-16T21:58:05.473 INFO:teuthology.orchestra.run.smithi122.stdout:46 2023-12-16T21:58:05.473 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45, 46] 2023-12-16T21:58:05.473 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:58:05.473 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2023-12-16T21:58:05.473 DEBUG:teuthology.orchestra.run.smithi122:> import re 2023-12-16T21:58:05.473 DEBUG:teuthology.orchestra.run.smithi122:> import os 2023-12-16T21:58:05.473 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2023-12-16T21:58:05.473 DEBUG:teuthology.orchestra.run.smithi122:> 2023-12-16T21:58:05.473 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2023-12-16T21:58:05.473 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2023-12-16T21:58:05.474 DEBUG:teuthology.orchestra.run.smithi122:> return f 2023-12-16T21:58:05.475 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:58:05.475 DEBUG:teuthology.orchestra.run.smithi122:> 2023-12-16T21:58:05.475 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.3")) 2023-12-16T21:58:05.475 DEBUG:teuthology.orchestra.run.smithi122:> ' 2023-12-16T21:58:05.597 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.3.44458.asok 2023-12-16T21:58:05.601 INFO:teuthology.orchestra.run.smithi122.stderr:2023-12-16T21:58:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:58:05.609 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44458.asok 2023-12-16T21:58:05.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:05.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44458.asok status 2023-12-16T21:58:05.762 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2023-12-16T21:58:05.762 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2023-12-16T21:58:05.762 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:58:05.762 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:58:05.762 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "3", 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "44458", 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 4454, 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 4454 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-12-16T21:58:05.763 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.2:0", 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 4121105056 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: } 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.2:0", 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 4121105056 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.4454 192.168.0.2:0/4121105056", 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.2:0/4121105056", 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 8, 2023-12-16T21:58:05.764 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 23, 2023-12-16T21:58:05.765 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:58:05.765 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2023-12-16T21:58:05.765 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2023-12-16T21:58:05.765 INFO:teuthology.orchestra.run.smithi122.stdout:} 2023-12-16T21:58:05.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:05.778 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2023-12-16T21:58:05.778 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:05.786 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2023-12-16T21:58:05.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:05.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:05.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:05.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:05.844 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2023-12-16T21:58:05.844 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.2 2023-12-16T21:58:05.893 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2023-12-16T21:58:05.893 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-12-16T21:58:05.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:05.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-12-16T21:58:05.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:05.953 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2023-12-16T21:58:05.953 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:05.960 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2023-12-16T21:58:05.960 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:05.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:05.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:06.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:06.017 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2023-12-16T21:58:06.017 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.3 2023-12-16T21:58:06.065 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2023-12-16T21:58:06.065 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-12-16T21:58:06.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:06.066 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-12-16T21:58:06.120 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-12-16T21:58:06.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e67f7640 1 Processor -- start 2023-12-16T21:58:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e67f7640 1 -- start start 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e67f7640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f93e013cf30 0x7f93e006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e67f7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93e015ad40 0x7f93e00cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e67f7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e015b170 con 0x7f93e006c800 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e67f7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e013d310 con 0x7f93e015ad40 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e67f7640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93e01404e0 con 0x7f93e013cf30 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e4d94640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93e015ad40 0x7f93e00cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e4d94640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93e015ad40 0x7f93e00cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38328/0 (socket says 172.21.15.28:38328) 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e4d94640 1 -- 172.21.15.28:0/4082139821 learned_addr learned my addr 172.21.15.28:0/4082139821 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93dffff640 1 --2- 172.21.15.28:0/4082139821 >> v2:172.21.15.122:3300/0 conn(0x7f93e013cf30 0x7f93e006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e4d94640 1 -- 172.21.15.28:0/4082139821 >> v2:172.21.15.122:3300/0 conn(0x7f93e013cf30 msgr2=0x7f93e006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e4d94640 1 --2- 172.21.15.28:0/4082139821 >> v2:172.21.15.122:3300/0 conn(0x7f93e013cf30 0x7f93e006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e4d94640 1 -- 172.21.15.28:0/4082139821 >> v1:172.21.15.122:6789/0 conn(0x7f93e006c800 legacy=0x7f93e015a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.271+0000 7f93e4d94640 1 -- 172.21.15.28:0/4082139821 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e015b3a0 con 0x7f93e015ad40 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 --2- 172.21.15.28:0/4082139821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e00cd880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f93e006b510 tx=0x7f93d801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ec86da7363320a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93deffd640 1 -- 172.21.15.28:0/4082139821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d80a56b0 con 0x7f93e015ad40 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93deffd640 1 -- 172.21.15.28:0/4082139821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d80a5810 con 0x7f93e015ad40 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93deffd640 1 -- 172.21.15.28:0/4082139821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d80a5ba0 con 0x7f93e015ad40 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- 172.21.15.28:0/4082139821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 msgr2=0x7f93e00cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 --2- 172.21.15.28:0/4082139821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e00cd880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f93e006b510 tx=0x7f93d801c900 comp rx=0 tx=0).stop 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- 172.21.15.28:0/4082139821 shutdown_connections 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 --2- 172.21.15.28:0/4082139821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e00cd880 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 --2- 172.21.15.28:0/4082139821 >> v2:172.21.15.122:3300/0 conn(0x7f93e013cf30 0x7f93e006c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- 172.21.15.28:0/4082139821 >> 172.21.15.28:0/4082139821 conn(0x7f93e006a530 msgr2=0x7f93e006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:06.276 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- 172.21.15.28:0/4082139821 shutdown_connections 2023-12-16T21:58:06.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- 172.21.15.28:0/4082139821 wait complete. 2023-12-16T21:58:06.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 Processor -- start 2023-12-16T21:58:06.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- start start 2023-12-16T21:58:06.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e006c800 0x7f93e0155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f93e013cf30 0x7f93e0167880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e0136840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e0136d80 con 0x7f93e015ad40 2023-12-16T21:58:06.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93e01673f0 con 0x7f93e006c800 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e67f7640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93e01667e0 con 0x7f93e013cf30 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e0136840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e0136840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38342/0 (socket says 172.21.15.28:38342) 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 -- 172.21.15.28:0/2915854231 learned_addr learned my addr 172.21.15.28:0/2915854231 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93dffff640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e006c800 0x7f93e0155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 -- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f93e013cf30 msgr2=0x7f93e0167880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93df7fe640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f93e013cf30 0x7f93e0167880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f93e013cf30 0x7f93e0167880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 -- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e006c800 msgr2=0x7f93e0155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e006c800 0x7f93e0155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 -- 172.21.15.28:0/2915854231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8009b30 con 0x7f93e015ad40 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93dffff640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e006c800 0x7f93e0155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.275+0000 7f93e4d94640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e0136840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f93d801a530 tx=0x7f93d80196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93dcff9640 1 -- 172.21.15.28:0/2915854231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d80b8670 con 0x7f93e015ad40 2023-12-16T21:58:06.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93dcff9640 1 -- 172.21.15.28:0/2915854231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d80b87d0 con 0x7f93e015ad40 2023-12-16T21:58:06.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e0166ac0 con 0x7f93e015ad40 2023-12-16T21:58:06.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93dcff9640 1 -- 172.21.15.28:0/2915854231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d80b8b60 con 0x7f93e015ad40 2023-12-16T21:58:06.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e0167080 con 0x7f93e015ad40 2023-12-16T21:58:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e0167560 con 0x7f93e015ad40 2023-12-16T21:58:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93dcff9640 1 -- 172.21.15.28:0/2915854231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f93d80b8cc0 con 0x7f93e015ad40 2023-12-16T21:58:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93dcff9640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f93c0066470 0x7f93c0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93dcff9640 1 -- 172.21.15.28:0/2915854231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93d811f770 con 0x7f93e015ad40 2023-12-16T21:58:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93dffff640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f93c0066470 0x7f93c0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.279+0000 7f93dffff640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f93c0066470 0x7f93c0068930 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009840 tx=0x7f93d0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.283+0000 7f93dcff9640 1 -- 172.21.15.28:0/2915854231 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f93d80eccf0 con 0x7f93e015ad40 2023-12-16T21:58:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.443+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93e0069440 con 0x7f93e015ad40 2023-12-16T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.443+0000 7f93dcff9640 1 -- 172.21.15.28:0/2915854231 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f93d8009510 con 0x7f93e015ad40 2023-12-16T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:06.446 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f93c0066470 msgr2=0x7f93c0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f93c0066470 0x7f93c0068930 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009840 tx=0x7f93d0008040 comp rx=0 tx=0).stop 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 msgr2=0x7f93e0136840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e0136840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f93d801a530 tx=0x7f93d80196a0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 shutdown_connections 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f93c0066470 0x7f93c0068930 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93e015ad40 0x7f93e0136840 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f93e013cf30 0x7f93e0167880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 --2- 172.21.15.28:0/2915854231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f93e006c800 0x7f93e0155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 >> 172.21.15.28:0/2915854231 conn(0x7f93e006a530 msgr2=0x7f93e013e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 shutdown_connections 2023-12-16T21:58:06.449 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.447+0000 7f93e67f7640 1 -- 172.21.15.28:0/2915854231 wait complete. 2023-12-16T21:58:06.466 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:06.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe0a1c30640 1 Processor -- start 2023-12-16T21:58:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe0a1c30640 1 -- start start 2023-12-16T21:58:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe0a1c30640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 0x7fe09c13d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe0a1c30640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe09c13f5f0 0x7fe09c0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe0a1c30640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09c13d2b0 con 0x7fe09c071d90 2023-12-16T21:58:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe0a1c30640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09c140520 con 0x7fe09c13f5f0 2023-12-16T21:58:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe0a1c30640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0cd7f0 con 0x7fe09c15adb0 2023-12-16T21:58:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe09bfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe09c13f5f0 0x7fe09c0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe09bfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe09c13f5f0 0x7fe09c0ce200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38344/0 (socket says 172.21.15.28:38344) 2023-12-16T21:58:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe09bfff640 1 -- 172.21.15.28:0/3044779290 learned_addr learned my addr 172.21.15.28:0/3044779290 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe09affd640 1 --2- 172.21.15.28:0/3044779290 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 0x7fe09c13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe09bfff640 1 -- 172.21.15.28:0/3044779290 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 msgr2=0x7fe09c13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe09bfff640 1 --2- 172.21.15.28:0/3044779290 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 0x7fe09c13d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe09bfff640 1 -- 172.21.15.28:0/3044779290 >> v1:172.21.15.122:6789/0 conn(0x7fe09c071d90 legacy=0x7fe09c072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.595+0000 7fe09bfff640 1 -- 172.21.15.28:0/3044779290 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c0cdd70 con 0x7fe09c13f5f0 2023-12-16T21:58:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe09bfff640 1 --2- 172.21.15.28:0/3044779290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c13f5f0 0x7fe09c0ce200 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe094009e50 tx=0x7fe09401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43357c1f4bfb097f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe09a7fc640 1 -- 172.21.15.28:0/3044779290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0940a5820 con 0x7fe09c13f5f0 2023-12-16T21:58:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe09a7fc640 1 -- 172.21.15.28:0/3044779290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0940a5980 con 0x7fe09c13f5f0 2023-12-16T21:58:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe09a7fc640 1 -- 172.21.15.28:0/3044779290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0940a5d10 con 0x7fe09c13f5f0 2023-12-16T21:58:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3044779290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c13f5f0 msgr2=0x7fe09c0ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3044779290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c13f5f0 0x7fe09c0ce200 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe094009e50 tx=0x7fe09401c9a0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3044779290 shutdown_connections 2023-12-16T21:58:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3044779290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c13f5f0 0x7fe09c0ce200 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3044779290 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 0x7fe09c13d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3044779290 >> 172.21.15.28:0/3044779290 conn(0x7fe09c0b0cc0 msgr2=0x7fe09c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3044779290 shutdown_connections 2023-12-16T21:58:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.599+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3044779290 wait complete. 2023-12-16T21:58:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 Processor -- start 2023-12-16T21:58:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 -- start start 2023-12-16T21:58:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c071d90 0x7fe09c1432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe09c13f5f0 0x7fe09c15ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 0x7fe09c1344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c141ec0 con 0x7fe09c071d90 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c137120 con 0x7fe09c13f5f0 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c140e40 con 0x7fe09c15adb0 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c071d90 0x7fe09c1432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c071d90 0x7fe09c1432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38356/0 (socket says 172.21.15.28:38356) 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 -- 172.21.15.28:0/3983023823 learned_addr learned my addr 172.21.15.28:0/3983023823 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09affd640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe09c13f5f0 0x7fe09c15ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 -- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 msgr2=0x7fe09c1344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09bfff640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 0x7fe09c1344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 0x7fe09c1344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 -- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe09c13f5f0 msgr2=0x7fe09c15ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.606 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe09c13f5f0 0x7fe09c15ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 -- 172.21.15.28:0/3983023823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe094009b30 con 0x7fe09c071d90 2023-12-16T21:58:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09affd640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe09c13f5f0 0x7fe09c15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe09b7fe640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c071d90 0x7fe09c1432a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c550 tx=0x7fe08c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe083fff640 1 -- 172.21.15.28:0/3983023823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c012830 con 0x7fe09c071d90 2023-12-16T21:58:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe083fff640 1 -- 172.21.15.28:0/3983023823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c012990 con 0x7fe09c071d90 2023-12-16T21:58:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09c141100 con 0x7fe09c071d90 2023-12-16T21:58:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe083fff640 1 -- 172.21.15.28:0/3983023823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c012d40 con 0x7fe09c071d90 2023-12-16T21:58:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09c1416c0 con 0x7fe09c071d90 2023-12-16T21:58:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe09c0693d0 con 0x7fe09c071d90 2023-12-16T21:58:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe083fff640 1 -- 172.21.15.28:0/3983023823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe08c01b400 con 0x7fe09c071d90 2023-12-16T21:58:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.603+0000 7fe083fff640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe0880664a0 0x7fe088068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.607+0000 7fe083fff640 1 -- 172.21.15.28:0/3983023823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe08c084070 con 0x7fe09c071d90 2023-12-16T21:58:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.607+0000 7fe09affd640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe0880664a0 0x7fe088068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.607+0000 7fe09affd640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe0880664a0 0x7fe088068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe09c13a510 tx=0x7fe090008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.611 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.611+0000 7fe083fff640 1 -- 172.21.15.28:0/3983023823 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe08c0514f0 con 0x7fe09c071d90 2023-12-16T21:58:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe09c06e710 con 0x7fe09c071d90 2023-12-16T21:58:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe083fff640 1 -- 172.21.15.28:0/3983023823 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe08c016020 con 0x7fe09c071d90 2023-12-16T21:58:06.773 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe0880664a0 msgr2=0x7fe088068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe0880664a0 0x7fe088068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe09c13a510 tx=0x7fe090008040 comp rx=0 tx=0).stop 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c071d90 msgr2=0x7fe09c1432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c071d90 0x7fe09c1432a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00c550 tx=0x7fe08c00ca10 comp rx=0 tx=0).stop 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 shutdown_connections 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe0880664a0 0x7fe088068960 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09c15adb0 0x7fe09c1344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe09c13f5f0 0x7fe09c15ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 --2- 172.21.15.28:0/3983023823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09c071d90 0x7fe09c1432a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 >> 172.21.15.28:0/3983023823 conn(0x7fe09c0b0cc0 msgr2=0x7fe09c0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 shutdown_connections 2023-12-16T21:58:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.771+0000 7fe0a1c30640 1 -- 172.21.15.28:0/3983023823 wait complete. 2023-12-16T21:58:06.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:06.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe444a9e640 1 Processor -- start 2023-12-16T21:58:06.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe444a9e640 1 -- start start 2023-12-16T21:58:06.909 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe444a9e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 0x7fe44013d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.909 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe444a9e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe44013f5f0 0x7fe4400ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.909 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe444a9e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44013d2b0 con 0x7fe440071d90 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe444a9e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe440140520 con 0x7fe44013f5f0 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe444a9e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4400cd7f0 con 0x7fe44015adb0 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43e59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe440071d90 0x7fe440072170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59034/0 (socket says 172.21.15.28:59034) 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43e59c640 1 -- 172.21.15.28:0/2388351832 learned_addr learned my addr 172.21.15.28:0/2388351832 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43ed9d640 1 --2- 172.21.15.28:0/2388351832 >> v2:172.21.15.28:3300/0 conn(0x7fe44013f5f0 0x7fe4400ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43dd9b640 1 --2- 172.21.15.28:0/2388351832 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 0x7fe44013d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43d59a640 1 -- 172.21.15.28:0/2388351832 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235460678 0 0) 0x7fe44013d2b0 con 0x7fe440071d90 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43d59a640 1 -- 172.21.15.28:0/2388351832 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe428005740 con 0x7fe440071d90 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43ed9d640 1 -- 172.21.15.28:0/2388351832 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 msgr2=0x7fe44013d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43ed9d640 1 --2- 172.21.15.28:0/2388351832 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 0x7fe44013d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43d59a640 1 -- 172.21.15.28:0/2388351832 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4062621506 0 0) 0x7fe428005740 con 0x7fe440071d90 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43ed9d640 1 -- 172.21.15.28:0/2388351832 >> v1:172.21.15.28:6789/0 conn(0x7fe440071d90 legacy=0x7fe440072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.907+0000 7fe43ed9d640 1 -- 172.21.15.28:0/2388351832 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4400cdd70 con 0x7fe44013f5f0 2023-12-16T21:58:06.911 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43ed9d640 1 --2- 172.21.15.28:0/2388351832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe4400ce200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe438009d90 tx=0x7fe43801c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee108fd74a68a120 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.911 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43d59a640 1 -- 172.21.15.28:0/2388351832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe430001680 con 0x7fe44013f5f0 2023-12-16T21:58:06.912 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43d59a640 1 -- 172.21.15.28:0/2388351832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4380a5540 con 0x7fe44013f5f0 2023-12-16T21:58:06.912 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43d59a640 1 -- 172.21.15.28:0/2388351832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4380a58b0 con 0x7fe44013f5f0 2023-12-16T21:58:06.912 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- 172.21.15.28:0/2388351832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 msgr2=0x7fe4400ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.912 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 --2- 172.21.15.28:0/2388351832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe4400ce200 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe438009d90 tx=0x7fe43801c840 comp rx=0 tx=0).stop 2023-12-16T21:58:06.912 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- 172.21.15.28:0/2388351832 shutdown_connections 2023-12-16T21:58:06.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 --2- 172.21.15.28:0/2388351832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe4400ce200 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 --2- 172.21.15.28:0/2388351832 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 0x7fe44013d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- 172.21.15.28:0/2388351832 >> 172.21.15.28:0/2388351832 conn(0x7fe4400b0cc0 msgr2=0x7fe4400b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:06.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- 172.21.15.28:0/2388351832 shutdown_connections 2023-12-16T21:58:06.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- 172.21.15.28:0/2388351832 wait complete. 2023-12-16T21:58:06.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 Processor -- start 2023-12-16T21:58:06.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- start start 2023-12-16T21:58:06.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe440071d90 0x7fe44014d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe440153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 0x7fe440154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe440137250 con 0x7fe44013f5f0 2023-12-16T21:58:06.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe440151ac0 con 0x7fe440071d90 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe440153220 con 0x7fe44015adb0 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe440153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe440153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38362/0 (socket says 172.21.15.28:38362) 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 -- 172.21.15.28:0/94901549 learned_addr learned my addr 172.21.15.28:0/94901549 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43e59c640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe440071d90 0x7fe44014d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 -- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 msgr2=0x7fe440154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 0x7fe440154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 -- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe440071d90 msgr2=0x7fe44014d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe440071d90 0x7fe44014d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 -- 172.21.15.28:0/94901549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe438009a70 con 0x7fe44013f5f0 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43e59c640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe440071d90 0x7fe44014d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe43dd9b640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe440153fe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe43400d800 tx=0x7fe43400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe42effd640 1 -- 172.21.15.28:0/94901549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43400b6c0 con 0x7fe44013f5f0 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe42effd640 1 -- 172.21.15.28:0/94901549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43400b820 con 0x7fe44013f5f0 2023-12-16T21:58:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe440153500 con 0x7fe44013f5f0 2023-12-16T21:58:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe42effd640 1 -- 172.21.15.28:0/94901549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43400bb90 con 0x7fe44013f5f0 2023-12-16T21:58:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.911+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe440153ac0 con 0x7fe44013f5f0 2023-12-16T21:58:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.915+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4400693d0 con 0x7fe44013f5f0 2023-12-16T21:58:06.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.915+0000 7fe42effd640 1 -- 172.21.15.28:0/94901549 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe43400bcf0 con 0x7fe44013f5f0 2023-12-16T21:58:06.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.915+0000 7fe42effd640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4200664a0 0x7fe420068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:06.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.915+0000 7fe42effd640 1 -- 172.21.15.28:0/94901549 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe434084e80 con 0x7fe44013f5f0 2023-12-16T21:58:06.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.915+0000 7fe43e59c640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4200664a0 0x7fe420068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:06.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.915+0000 7fe43e59c640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4200664a0 0x7fe420068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe430007a10 tx=0x7fe430009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:06.919+0000 7fe42effd640 1 -- 172.21.15.28:0/94901549 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe434052370 con 0x7fe44013f5f0 2023-12-16T21:58:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.071+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe44006e710 con 0x7fe44013f5f0 2023-12-16T21:58:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.071+0000 7fe42effd640 1 -- 172.21.15.28:0/94901549 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe434055990 con 0x7fe44013f5f0 2023-12-16T21:58:07.075 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4200664a0 msgr2=0x7fe420068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4200664a0 0x7fe420068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe430007a10 tx=0x7fe430009040 comp rx=0 tx=0).stop 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 msgr2=0x7fe440153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe440153fe0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe43400d800 tx=0x7fe43400dcc0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 shutdown_connections 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4200664a0 0x7fe420068960 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe44015adb0 0x7fe440154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe44013f5f0 0x7fe440153fe0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 --2- 172.21.15.28:0/94901549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe440071d90 0x7fe44014d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 >> 172.21.15.28:0/94901549 conn(0x7fe4400b0cc0 msgr2=0x7fe44013ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 shutdown_connections 2023-12-16T21:58:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.075+0000 7fe444a9e640 1 -- 172.21.15.28:0/94901549 wait complete. 2023-12-16T21:58:07.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:07.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a48c71640 1 Processor -- start 2023-12-16T21:58:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a48c71640 1 -- start start 2023-12-16T21:58:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a48c71640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415d690 0x7f4a440cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a48c71640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440d0570 con 0x7f4a440d0120 2023-12-16T21:58:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a48c71640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440aff30 con 0x7f4a4415d2b0 2023-12-16T21:58:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a48c71640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440d1ec0 con 0x7f4a4415d690 2023-12-16T21:58:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a42d9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4a440d0120 0x7f4a440b0cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59036/0 (socket says 172.21.15.28:59036) 2023-12-16T21:58:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a42d9d640 1 -- 172.21.15.28:0/3075819473 learned_addr learned my addr 172.21.15.28:0/3075819473 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a41d9b640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415d690 0x7f4a440cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391542194 0 0) 0x7f4a34001120 con 0x7f4a440d0120 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a2c003530 con 0x7f4a440d0120 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2332526324 0 0) 0x7f4a2c003530 con 0x7f4a440d0120 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415d690 msgr2=0x7f4a440cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415d690 0x7f4a440cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 >> v1:172.21.15.122:6789/0 conn(0x7f4a4415d2b0 legacy=0x7f4a440d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440b01d0 con 0x7f4a440d0120 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f4a34001810 con 0x7f4a440d0120 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 >> v1:172.21.15.28:6789/0 conn(0x7f4a440d0120 legacy=0x7f4a440b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a2c0047d0 0x7f4a2c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415d690 0x7f4a2c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440d1ec0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c00ba40 con 0x7f4a2c0047d0 2023-12-16T21:58:07.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c00bc00 con 0x7f4a4415d690 2023-12-16T21:58:07.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a4259c640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a42d9d640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415d690 0x7f4a2c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.219+0000 7f4a41d9b640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a2c0047d0 0x7f4a2c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4259c640 1 -- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415d690 msgr2=0x7f4a2c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:07.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4259c640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415d690 0x7f4a2c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4259c640 1 -- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a2c0047d0 msgr2=0x7f4a2c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4259c640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a2c0047d0 0x7f4a2c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4259c640 1 -- 172.21.15.28:0/3075819473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c003530 con 0x7f4a2c0039b0 2023-12-16T21:58:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4259c640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a2c003dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c001e40 tx=0x7f4a3c01c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7fa9f376def3d0b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c003b80 con 0x7f4a2c0039b0 2023-12-16T21:58:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c003ce0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a4159a640 1 -- 172.21.15.28:0/3075819473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c005310 con 0x7f4a2c0039b0 2023-12-16T21:58:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a48c71640 1 -- 172.21.15.28:0/3075819473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 msgr2=0x7f4a2c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a48c71640 1 --2- 172.21.15.28:0/3075819473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a2c003dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c001e40 tx=0x7f4a3c01c6f0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a42d9d640 1 -- 172.21.15.28:0/3075819473 reap_dead start 2023-12-16T21:58:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a48c71640 1 -- 172.21.15.28:0/3075819473 shutdown_connections 2023-12-16T21:58:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a48c71640 1 -- 172.21.15.28:0/3075819473 >> 172.21.15.28:0/3075819473 conn(0x7f4a4409c510 msgr2=0x7f4a440d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a48c71640 1 -- 172.21.15.28:0/3075819473 shutdown_connections 2023-12-16T21:58:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.223+0000 7f4a48c71640 1 -- 172.21.15.28:0/3075819473 wait complete. 2023-12-16T21:58:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 Processor -- start 2023-12-16T21:58:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 -- start start 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a441638d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a440cc890 0x7f4a440ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415ac30 0x7f4a4415b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4415b5f0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a44163e10 con 0x7f4a440cc890 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a44161820 con 0x7f4a4415ac30 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a441638d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a441638d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38384/0 (socket says 172.21.15.28:38384) 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 -- 172.21.15.28:0/3003710430 learned_addr learned my addr 172.21.15.28:0/3003710430 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a41d9b640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a440cc890 0x7f4a440ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a42d9d640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415ac30 0x7f4a4415b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 -- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415ac30 msgr2=0x7f4a4415b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415ac30 0x7f4a4415b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 -- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a440cc890 msgr2=0x7f4a440ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a440cc890 0x7f4a440ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 -- 172.21.15.28:0/3003710430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440ca5d0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a41d9b640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a440cc890 0x7f4a440ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a4259c640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a441638d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c097ff0 tx=0x7f4a2c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a32ffd640 1 -- 172.21.15.28:0/3003710430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c003b80 con 0x7f4a2c0039b0 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a32ffd640 1 -- 172.21.15.28:0/3003710430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c0050d0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a32ffd640 1 -- 172.21.15.28:0/3003710430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c005480 con 0x7f4a2c0039b0 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a440ca8b0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a440cae70 con 0x7f4a2c0039b0 2023-12-16T21:58:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.227+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a440679f0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.231+0000 7f4a32ffd640 1 -- 172.21.15.28:0/3003710430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4a3c0055e0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.231+0000 7f4a32ffd640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4a240664a0 0x7f4a24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.231+0000 7f4a32ffd640 1 -- 172.21.15.28:0/3003710430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a3c11e530 con 0x7f4a2c0039b0 2023-12-16T21:58:07.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.231+0000 7f4a41d9b640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4a240664a0 0x7f4a24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.231+0000 7f4a41d9b640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4a240664a0 0x7f4a24068960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4a44160a70 tx=0x7f4a38008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.235+0000 7f4a32ffd640 1 -- 172.21.15.28:0/3003710430 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4a3c0ebb70 con 0x7f4a2c0039b0 2023-12-16T21:58:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.383+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4a440693d0 con 0x7f4a2c0039b0 2023-12-16T21:58:07.387 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.383+0000 7f4a32ffd640 1 -- 172.21.15.28:0/3003710430 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4a3c0ef190 con 0x7f4a2c0039b0 2023-12-16T21:58:07.387 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4a240664a0 msgr2=0x7f4a24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4a240664a0 0x7f4a24068960 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4a44160a70 tx=0x7f4a38008040 comp rx=0 tx=0).stop 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 msgr2=0x7f4a441638d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a441638d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c097ff0 tx=0x7f4a2c005800 comp rx=0 tx=0).stop 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 shutdown_connections 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4a240664a0 0x7f4a24068960 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4a4415ac30 0x7f4a4415b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4a440cc890 0x7f4a440ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 --2- 172.21.15.28:0/3003710430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a2c0039b0 0x7f4a441638d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 >> 172.21.15.28:0/3003710430 conn(0x7f4a4409c510 msgr2=0x7f4a440d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 shutdown_connections 2023-12-16T21:58:07.390 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.387+0000 7f4a48c71640 1 -- 172.21.15.28:0/3003710430 wait complete. 2023-12-16T21:58:07.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5977612640 1 Processor -- start 2023-12-16T21:58:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5977612640 1 -- start start 2023-12-16T21:58:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5977612640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5970146b90 0x7f597015e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5977612640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f597006a530 0x7f597006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5977612640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5970161220 con 0x7f59700b4f90 2023-12-16T21:58:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5977612640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597006e6b0 con 0x7f597006a530 2023-12-16T21:58:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5977612640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597006af20 con 0x7f5970146b90 2023-12-16T21:58:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5975baf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f597006a530 0x7f597006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5975baf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f597006a530 0x7f597006a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60150/0 (socket says 172.21.15.28:60150) 2023-12-16T21:58:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5975baf640 1 -- 172.21.15.28:0/1529992419 learned_addr learned my addr 172.21.15.28:0/1529992419 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f59753ae640 1 --2- 172.21.15.28:0/1529992419 >> v2:172.21.15.122:3300/0 conn(0x7f5970146b90 0x7f597015e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5975baf640 1 -- 172.21.15.28:0/1529992419 >> v2:172.21.15.122:3300/0 conn(0x7f5970146b90 msgr2=0x7f597015e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5975baf640 1 --2- 172.21.15.28:0/1529992419 >> v2:172.21.15.122:3300/0 conn(0x7f5970146b90 0x7f597015e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5975baf640 1 -- 172.21.15.28:0/1529992419 >> v1:172.21.15.122:6789/0 conn(0x7f59700b4f90 legacy=0x7f59701616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.531+0000 7f5975baf640 1 -- 172.21.15.28:0/1529992419 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597015e750 con 0x7f597006a530 2023-12-16T21:58:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5975baf640 1 --2- 172.21.15.28:0/1529992419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597006a530 0x7f597006a9e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f59700b2a60 tx=0x7f596801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac69a353e51ff053 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5967fff640 1 -- 172.21.15.28:0/1529992419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59680a55c0 con 0x7f597006a530 2023-12-16T21:58:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5967fff640 1 -- 172.21.15.28:0/1529992419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59680a5720 con 0x7f597006a530 2023-12-16T21:58:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5967fff640 1 -- 172.21.15.28:0/1529992419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59680a5ab0 con 0x7f597006a530 2023-12-16T21:58:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- 172.21.15.28:0/1529992419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597006a530 msgr2=0x7f597006a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 --2- 172.21.15.28:0/1529992419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597006a530 0x7f597006a9e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f59700b2a60 tx=0x7f596801c900 comp rx=0 tx=0).stop 2023-12-16T21:58:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- 172.21.15.28:0/1529992419 shutdown_connections 2023-12-16T21:58:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 --2- 172.21.15.28:0/1529992419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597006a530 0x7f597006a9e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 --2- 172.21.15.28:0/1529992419 >> v2:172.21.15.122:3300/0 conn(0x7f5970146b90 0x7f597015e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- 172.21.15.28:0/1529992419 >> 172.21.15.28:0/1529992419 conn(0x7f59700b1bd0 msgr2=0x7f59700b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- 172.21.15.28:0/1529992419 shutdown_connections 2023-12-16T21:58:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- 172.21.15.28:0/1529992419 wait complete. 2023-12-16T21:58:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 Processor -- start 2023-12-16T21:58:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- start start 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f597006a530 0x7f597015dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59700b4f90 0x7f5970135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5970146b90 0x7f597013aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59701352d0 con 0x7f59700b4f90 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59701346c0 con 0x7f597006a530 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.535+0000 7f5977612640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59701349a0 con 0x7f5970146b90 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59700b4f90 0x7f5970135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59700b4f90 0x7f5970135c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60152/0 (socket says 172.21.15.28:60152) 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 -- 172.21.15.28:0/961293806 learned_addr learned my addr 172.21.15.28:0/961293806 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f59753ae640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f597006a530 0x7f597015dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5975baf640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5970146b90 0x7f597013aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 -- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5970146b90 msgr2=0x7f597013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5970146b90 0x7f597013aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 -- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f597006a530 msgr2=0x7f597015dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f597006a530 0x7f597015dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 -- 172.21.15.28:0/961293806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968009b30 con 0x7f59700b4f90 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5974bad640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59700b4f90 0x7f5970135c70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f596000c450 tx=0x7f596000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5965ffb640 1 -- 172.21.15.28:0/961293806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960012830 con 0x7f59700b4f90 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5965ffb640 1 -- 172.21.15.28:0/961293806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59600129b0 con 0x7f59700b4f90 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5965ffb640 1 -- 172.21.15.28:0/961293806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960012d60 con 0x7f59700b4f90 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5970134c80 con 0x7f59700b4f90 2023-12-16T21:58:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5970135760 con 0x7f59700b4f90 2023-12-16T21:58:07.542 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5970135440 con 0x7f59700b4f90 2023-12-16T21:58:07.542 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5965ffb640 1 -- 172.21.15.28:0/961293806 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f596001c460 con 0x7f59700b4f90 2023-12-16T21:58:07.542 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5965ffb640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5954066470 0x7f5954068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.542 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f5965ffb640 1 -- 172.21.15.28:0/961293806 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5960083ce0 con 0x7f59700b4f90 2023-12-16T21:58:07.542 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f59753ae640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5954066470 0x7f5954068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.543 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.539+0000 7f59753ae640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5954066470 0x7f5954068930 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5970143d90 tx=0x7f596c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.543+0000 7f5965ffb640 1 -- 172.21.15.28:0/961293806 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f59600511f0 con 0x7f59700b4f90 2023-12-16T21:58:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.703+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59700674d0 con 0x7f59700b4f90 2023-12-16T21:58:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5965ffb640 1 -- 172.21.15.28:0/961293806 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f5960054810 con 0x7f59700b4f90 2023-12-16T21:58:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:07.708 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5954066470 msgr2=0x7f5954068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5954066470 0x7f5954068930 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5970143d90 tx=0x7f596c008040 comp rx=0 tx=0).stop 2023-12-16T21:58:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59700b4f90 msgr2=0x7f5970135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59700b4f90 0x7f5970135c70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f596000c450 tx=0x7f596000c910 comp rx=0 tx=0).stop 2023-12-16T21:58:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 shutdown_connections 2023-12-16T21:58:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5954066470 0x7f5954068930 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5970146b90 0x7f597013aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59700b4f90 0x7f5970135c70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 --2- 172.21.15.28:0/961293806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f597006a530 0x7f597015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 >> 172.21.15.28:0/961293806 conn(0x7f59700b1bd0 msgr2=0x7f597013fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 shutdown_connections 2023-12-16T21:58:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.707+0000 7f5977612640 1 -- 172.21.15.28:0/961293806 wait complete. 2023-12-16T21:58:07.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:07.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:07.855 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.851+0000 7f9d791a1640 1 Processor -- start 2023-12-16T21:58:07.855 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.851+0000 7f9d791a1640 1 -- start start 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d791a1640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9d741448f0 0x7f9d740ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d791a1640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d740b1fa0 0x7f9d740b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d791a1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d740cd4c0 con 0x7f9d7415b880 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d791a1640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d740b2910 con 0x7f9d741448f0 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d791a1640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7415bd40 con 0x7f9d740b1fa0 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d72d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9d741448f0 0x7f9d740ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d72d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9d741448f0 0x7f9d740ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60156/0 (socket says 172.21.15.28:60156) 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d72d9d640 1 -- 172.21.15.28:0/981355411 learned_addr learned my addr 172.21.15.28:0/981355411 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d7359e640 1 --2- 172.21.15.28:0/981355411 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d740b1fa0 0x7f9d740b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d72d9d640 1 -- 172.21.15.28:0/981355411 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d740b1fa0 msgr2=0x7f9d740b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d72d9d640 1 --2- 172.21.15.28:0/981355411 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d740b1fa0 0x7f9d740b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d72d9d640 1 -- 172.21.15.28:0/981355411 >> v1:172.21.15.122:6789/0 conn(0x7f9d7415b880 legacy=0x7f9d740ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d72d9d640 1 -- 172.21.15.28:0/981355411 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7415c020 con 0x7f9d741448f0 2023-12-16T21:58:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d7359e640 1 --2- 172.21.15.28:0/981355411 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d740b1fa0 0x7f9d740b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:07.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d72d9d640 1 --2- 172.21.15.28:0/981355411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d741448f0 0x7f9d740ccf80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009870 tx=0x7f9d6401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f509d1d75b3bbc47 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d71d9b640 1 -- 172.21.15.28:0/981355411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d640ab600 con 0x7f9d741448f0 2023-12-16T21:58:07.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d71d9b640 1 -- 172.21.15.28:0/981355411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d640a7050 con 0x7f9d741448f0 2023-12-16T21:58:07.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.855+0000 7f9d71d9b640 1 -- 172.21.15.28:0/981355411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d640ab980 con 0x7f9d741448f0 2023-12-16T21:58:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- 172.21.15.28:0/981355411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d741448f0 msgr2=0x7f9d740ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 --2- 172.21.15.28:0/981355411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d741448f0 0x7f9d740ccf80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009870 tx=0x7f9d6401c470 comp rx=0 tx=0).stop 2023-12-16T21:58:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- 172.21.15.28:0/981355411 shutdown_connections 2023-12-16T21:58:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 --2- 172.21.15.28:0/981355411 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d740b1fa0 0x7f9d740b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 --2- 172.21.15.28:0/981355411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d741448f0 0x7f9d740ccf80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- 172.21.15.28:0/981355411 >> 172.21.15.28:0/981355411 conn(0x7f9d740b54d0 msgr2=0x7f9d740b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- 172.21.15.28:0/981355411 shutdown_connections 2023-12-16T21:58:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- 172.21.15.28:0/981355411 wait complete. 2023-12-16T21:58:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 Processor -- start 2023-12-16T21:58:07.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- start start 2023-12-16T21:58:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d740b1fa0 0x7f9d7413e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d741448f0 0x7f9d7413f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d7415b880 0x7f9d741f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d741f6ae0 con 0x7f9d740b1fa0 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d74161550 con 0x7f9d741448f0 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d791a1640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7413eba0 con 0x7f9d7415b880 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d740b1fa0 0x7f9d7413e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d740b1fa0 0x7f9d7413e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60158/0 (socket says 172.21.15.28:60158) 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 -- 172.21.15.28:0/3781480198 learned_addr learned my addr 172.21.15.28:0/3781480198 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 -- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d7415b880 msgr2=0x7f9d741f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d7259c640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d741448f0 0x7f9d7413f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d7415b880 0x7f9d741f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 -- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d741448f0 msgr2=0x7f9d7413f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d741448f0 0x7f9d7413f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 -- 172.21.15.28:0/3781480198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640094d0 con 0x7f9d740b1fa0 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.859+0000 7f9d72d9d640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d740b1fa0 0x7f9d7413e660 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9d640ae040 tx=0x7f9d64019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d5b7fe640 1 -- 172.21.15.28:0/3781480198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d640b4640 con 0x7f9d740b1fa0 2023-12-16T21:58:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d5b7fe640 1 -- 172.21.15.28:0/3781480198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d640b47c0 con 0x7f9d740b1fa0 2023-12-16T21:58:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d5b7fe640 1 -- 172.21.15.28:0/3781480198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d640b4b70 con 0x7f9d740b1fa0 2023-12-16T21:58:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d741f6520 con 0x7f9d740b1fa0 2023-12-16T21:58:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d741611e0 con 0x7f9d740b1fa0 2023-12-16T21:58:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d740cc7e0 con 0x7f9d740b1fa0 2023-12-16T21:58:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d5b7fe640 1 -- 172.21.15.28:0/3781480198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f9d640b2070 con 0x7f9d740b1fa0 2023-12-16T21:58:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d5b7fe640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9d60066470 0x7f9d60068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d5b7fe640 1 -- 172.21.15.28:0/3781480198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d6411de30 con 0x7f9d740b1fa0 2023-12-16T21:58:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d7259c640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9d60066470 0x7f9d60068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.863+0000 7f9d7259c640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9d60066470 0x7f9d60068930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9d74165fc0 tx=0x7f9d68008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:07.869 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:07.867+0000 7f9d5b7fe640 1 -- 172.21.15.28:0/3781480198 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f9d640eb340 con 0x7f9d740b1fa0 2023-12-16T21:58:08.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.023+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d740674d0 con 0x7f9d740b1fa0 2023-12-16T21:58:08.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.023+0000 7f9d5b7fe640 1 -- 172.21.15.28:0/3781480198 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d640ee960 con 0x7f9d740b1fa0 2023-12-16T21:58:08.025 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9d60066470 msgr2=0x7f9d60068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9d60066470 0x7f9d60068930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9d74165fc0 tx=0x7f9d68008040 comp rx=0 tx=0).stop 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d740b1fa0 msgr2=0x7f9d7413e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d740b1fa0 0x7f9d7413e660 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9d640ae040 tx=0x7f9d64019210 comp rx=0 tx=0).stop 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 shutdown_connections 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9d60066470 0x7f9d60068930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d7415b880 0x7f9d741f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d741448f0 0x7f9d7413f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 --2- 172.21.15.28:0/3781480198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d740b1fa0 0x7f9d7413e660 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 >> 172.21.15.28:0/3781480198 conn(0x7f9d740b54d0 msgr2=0x7f9d740712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 shutdown_connections 2023-12-16T21:58:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.027+0000 7f9d791a1640 1 -- 172.21.15.28:0/3781480198 wait complete. 2023-12-16T21:58:08.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:08.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f1024e0e640 1 Processor -- start 2023-12-16T21:58:08.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f1024e0e640 1 -- start start 2023-12-16T21:58:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f1024e0e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015adb0 0x7f102013d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f1024e0e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f102013f5f0 0x7f10200ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f1024e0e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102013d2b0 con 0x7f1020071d90 2023-12-16T21:58:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f1024e0e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1020140520 con 0x7f102013f5f0 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f1024e0e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10200cd7f0 con 0x7f102015adb0 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101e59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1020071d90 0x7f1020072170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43818/0 (socket says 172.21.15.28:43818) 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101e59c640 1 -- 172.21.15.28:0/3541440876 learned_addr learned my addr 172.21.15.28:0/3541440876 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101dd9b640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015adb0 0x7f102013d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101ed9d640 1 --2- 172.21.15.28:0/3541440876 >> v2:172.21.15.122:3300/0 conn(0x7f102013f5f0 0x7f10200ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132416800 0 0) 0x7f102013d2b0 con 0x7f1020071d90 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1008003530 con 0x7f1020071d90 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3207259503 0 0) 0x7f1008003530 con 0x7f1020071d90 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015adb0 msgr2=0x7f102013d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015adb0 0x7f102013d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 >> v2:172.21.15.122:3300/0 conn(0x7f102013f5f0 msgr2=0x7f10200ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 --2- 172.21.15.28:0/3541440876 >> v2:172.21.15.122:3300/0 conn(0x7f102013f5f0 0x7f10200ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10200cdd70 con 0x7f1020071d90 2023-12-16T21:58:08.179 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f1010001680 con 0x7f1020071d90 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 >> v1:172.21.15.28:6789/0 conn(0x7f1020071d90 legacy=0x7f1020072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f1008003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10080047d0 0x7f1008004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015adb0 0x7f1008005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1020140520 con 0x7f1008003980 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10200cd7f0 con 0x7f10080047d0 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100800dba0 con 0x7f102015adb0 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101e59c640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f1008003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101ed9d640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015adb0 0x7f1008005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101dd9b640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10080047d0 0x7f1008004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.180 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101e59c640 1 -- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015adb0 msgr2=0x7f1008005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101e59c640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015adb0 0x7f1008005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101e59c640 1 -- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10080047d0 msgr2=0x7f1008004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101e59c640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10080047d0 0x7f1008004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101e59c640 1 -- 172.21.15.28:0/3541440876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1008003530 con 0x7f1008003980 2023-12-16T21:58:08.181 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.175+0000 7f101dd9b640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10080047d0 0x7f1008004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:08.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f101e59c640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f1008003da0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f101000c290 tx=0x7f101001ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4496935a60eef2f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10100058e0 con 0x7f1008003980 2023-12-16T21:58:08.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010005a40 con 0x7f1008003980 2023-12-16T21:58:08.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f101d59a640 1 -- 172.21.15.28:0/3541440876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010005df0 con 0x7f1008003980 2023-12-16T21:58:08.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- 172.21.15.28:0/3541440876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 msgr2=0x7f1008003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 --2- 172.21.15.28:0/3541440876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f1008003da0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f101000c290 tx=0x7f101001ccb0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f101ed9d640 1 -- 172.21.15.28:0/3541440876 reap_dead start 2023-12-16T21:58:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- 172.21.15.28:0/3541440876 shutdown_connections 2023-12-16T21:58:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- 172.21.15.28:0/3541440876 >> 172.21.15.28:0/3541440876 conn(0x7f10200b0cc0 msgr2=0x7f10200b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- 172.21.15.28:0/3541440876 shutdown_connections 2023-12-16T21:58:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- 172.21.15.28:0/3541440876 wait complete. 2023-12-16T21:58:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 Processor -- start 2023-12-16T21:58:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- start start 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f10201369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f102013b9f0 0x7f102013be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015ea70 0x7f102015eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1020160420 con 0x7f1008003980 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102013c390 con 0x7f102013b9f0 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f1024e0e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102015f7a0 con 0x7f102015ea70 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.179+0000 7f101e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f10201369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f10201369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60174/0 (socket says 172.21.15.28:60174) 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101e59c640 1 -- 172.21.15.28:0/214314873 learned_addr learned my addr 172.21.15.28:0/214314873 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101ed9d640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015ea70 0x7f102015eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101dd9b640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f102013b9f0 0x7f102013be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101e59c640 1 -- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015ea70 msgr2=0x7f102015eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101e59c640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015ea70 0x7f102015eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101e59c640 1 -- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f102013b9f0 msgr2=0x7f102013be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101e59c640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f102013b9f0 0x7f102013be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101e59c640 1 -- 172.21.15.28:0/214314873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102015fa80 con 0x7f1008003980 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101e59c640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f10201369f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f101000c290 tx=0x7f1008005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f100effd640 1 -- 172.21.15.28:0/214314873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10100047f0 con 0x7f1008003980 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f100effd640 1 -- 172.21.15.28:0/214314873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010004970 con 0x7f1008003980 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102015fd60 con 0x7f1008003980 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f100effd640 1 -- 172.21.15.28:0/214314873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010004d20 con 0x7f1008003980 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102015f430 con 0x7f1008003980 2023-12-16T21:58:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10200693d0 con 0x7f1008003980 2023-12-16T21:58:08.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f100effd640 1 -- 172.21.15.28:0/214314873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1010007070 con 0x7f1008003980 2023-12-16T21:58:08.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f100effd640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1000066470 0x7f1000068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f100effd640 1 -- 172.21.15.28:0/214314873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f101011d820 con 0x7f1008003980 2023-12-16T21:58:08.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101dd9b640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1000066470 0x7f1000068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.183+0000 7f101dd9b640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1000066470 0x7f1000068930 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f10200cc4f0 tx=0x7f1014008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.187+0000 7f100effd640 1 -- 172.21.15.28:0/214314873 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f10100ead30 con 0x7f1008003980 2023-12-16T21:58:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.343+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f102013cd40 con 0x7f1008003980 2023-12-16T21:58:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f100effd640 1 -- 172.21.15.28:0/214314873 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f101011d070 con 0x7f1008003980 2023-12-16T21:58:08.348 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1000066470 msgr2=0x7f1000068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1000066470 0x7f1000068930 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f10200cc4f0 tx=0x7f1014008040 comp rx=0 tx=0).stop 2023-12-16T21:58:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 msgr2=0x7f10201369f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f10201369f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f101000c290 tx=0x7f1008005800 comp rx=0 tx=0).stop 2023-12-16T21:58:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 shutdown_connections 2023-12-16T21:58:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1000066470 0x7f1000068930 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f102015ea70 0x7f102015eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f102013b9f0 0x7f102013be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 --2- 172.21.15.28:0/214314873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1008003980 0x7f10201369f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 >> 172.21.15.28:0/214314873 conn(0x7f10200b0cc0 msgr2=0x7f10200b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 shutdown_connections 2023-12-16T21:58:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.347+0000 7f1024e0e640 1 -- 172.21.15.28:0/214314873 wait complete. 2023-12-16T21:58:08.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:08.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f6713393640 1 Processor -- start 2023-12-16T21:58:08.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f6713393640 1 -- start start 2023-12-16T21:58:08.484 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f6713393640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f670c15c260 0x7f670c0b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.484 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f6713393640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 0x7f670c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.484 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f6713393640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670c0b4720 con 0x7f670c0b3cd0 2023-12-16T21:58:08.484 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f6713393640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670c0b4a00 con 0x7f670c15c260 2023-12-16T21:58:08.485 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f6713393640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c0b2870 con 0x7f670c06bc50 2023-12-16T21:58:08.485 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f671112f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f670c15c260 0x7f670c0b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.485 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f6711930640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 0x7f670c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f671092e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f670c0b3cd0 0x7f670c06b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:55956/0 (socket says 172.21.15.28:55956) 2023-12-16T21:58:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f671092e640 1 -- 172.21.15.28:0/901163200 learned_addr learned my addr 172.21.15.28:0/901163200 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f671112f640 1 -- 172.21.15.28:0/901163200 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 msgr2=0x7f670c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f671112f640 1 --2- 172.21.15.28:0/901163200 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 0x7f670c0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f671112f640 1 -- 172.21.15.28:0/901163200 >> v1:172.21.15.122:6789/0 conn(0x7f670c0b3cd0 legacy=0x7f670c06b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.486 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.483+0000 7f671112f640 1 -- 172.21.15.28:0/901163200 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c0b2b50 con 0x7f670c15c260 2023-12-16T21:58:08.489 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f671112f640 1 --2- 172.21.15.28:0/901163200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c15c260 0x7f670c0b3070 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67080097c0 tx=0x7f670801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdbdc39493f2e9f4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.489 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f66fffff640 1 -- 172.21.15.28:0/901163200 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67080ab600 con 0x7f670c15c260 2023-12-16T21:58:08.489 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f66fffff640 1 -- 172.21.15.28:0/901163200 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67080a7050 con 0x7f670c15c260 2023-12-16T21:58:08.489 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f66fffff640 1 -- 172.21.15.28:0/901163200 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67080ab980 con 0x7f670c15c260 2023-12-16T21:58:08.490 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f6713393640 1 -- 172.21.15.28:0/901163200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c15c260 msgr2=0x7f670c0b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.490 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f6713393640 1 --2- 172.21.15.28:0/901163200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c15c260 0x7f670c0b3070 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67080097c0 tx=0x7f670801c480 comp rx=0 tx=0).stop 2023-12-16T21:58:08.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f6713393640 1 -- 172.21.15.28:0/901163200 shutdown_connections 2023-12-16T21:58:08.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f6713393640 1 --2- 172.21.15.28:0/901163200 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 0x7f670c0cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f6713393640 1 --2- 172.21.15.28:0/901163200 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c15c260 0x7f670c0b3070 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f6713393640 1 -- 172.21.15.28:0/901163200 >> 172.21.15.28:0/901163200 conn(0x7f670c0710b0 msgr2=0x7f670c0714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:08.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f6713393640 1 -- 172.21.15.28:0/901163200 shutdown_connections 2023-12-16T21:58:08.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.487+0000 7f6713393640 1 -- 172.21.15.28:0/901163200 wait complete. 2023-12-16T21:58:08.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 Processor -- start 2023-12-16T21:58:08.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 -- start start 2023-12-16T21:58:08.493 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 0x7f670c141af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.493 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c0b3cd0 0x7f670c153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f670c15c260 0x7f670c1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c1f7570 con 0x7f670c15c260 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c142030 con 0x7f670c0b3cd0 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670c157060 con 0x7f670c06bc50 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f670c15c260 0x7f670c1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f670c15c260 0x7f670c1541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60184/0 (socket says 172.21.15.28:60184) 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 -- 172.21.15.28:0/3286195330 learned_addr learned my addr 172.21.15.28:0/3286195330 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f671092e640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c0b3cd0 0x7f670c153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f671112f640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 0x7f670c141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 -- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 msgr2=0x7f670c141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 0x7f670c141af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 -- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c0b3cd0 msgr2=0x7f670c153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c0b3cd0 0x7f670c153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 -- 172.21.15.28:0/3286195330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708009420 con 0x7f670c15c260 2023-12-16T21:58:08.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f671092e640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c0b3cd0 0x7f670c153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:08.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6711930640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f670c15c260 0x7f670c1541e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6704009e30 tx=0x7f670400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f66fdffb640 1 -- 172.21.15.28:0/3286195330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704009ca0 con 0x7f670c15c260 2023-12-16T21:58:08.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f66fdffb640 1 -- 172.21.15.28:0/3286195330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670400f040 con 0x7f670c15c260 2023-12-16T21:58:08.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670c142540 con 0x7f670c15c260 2023-12-16T21:58:08.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f66fdffb640 1 -- 172.21.15.28:0/3286195330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67040146c0 con 0x7f670c15c260 2023-12-16T21:58:08.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670c156ae0 con 0x7f670c15c260 2023-12-16T21:58:08.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670c0674d0 con 0x7f670c15c260 2023-12-16T21:58:08.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f66fdffb640 1 -- 172.21.15.28:0/3286195330 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6704014820 con 0x7f670c15c260 2023-12-16T21:58:08.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.491+0000 7f66fdffb640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66f0066470 0x7f66f0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.495+0000 7f66fdffb640 1 -- 172.21.15.28:0/3286195330 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6704084e30 con 0x7f670c15c260 2023-12-16T21:58:08.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.495+0000 7f671112f640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66f0066470 0x7f66f0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.495+0000 7f671112f640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66f0066470 0x7f66f0068930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f67080ad000 tx=0x7f6708008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.499 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.499+0000 7f66fdffb640 1 -- 172.21.15.28:0/3286195330 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6704052210 con 0x7f670c15c260 2023-12-16T21:58:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.651+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f670c06f480 con 0x7f670c15c260 2023-12-16T21:58:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.651+0000 7f66fdffb640 1 -- 172.21.15.28:0/3286195330 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f67040180a0 con 0x7f670c15c260 2023-12-16T21:58:08.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66f0066470 msgr2=0x7f66f0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66f0066470 0x7f66f0068930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f67080ad000 tx=0x7f6708008040 comp rx=0 tx=0).stop 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f670c15c260 msgr2=0x7f670c1541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f670c15c260 0x7f670c1541e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6704009e30 tx=0x7f670400c490 comp rx=0 tx=0).stop 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 shutdown_connections 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66f0066470 0x7f66f0068930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f670c15c260 0x7f670c1541e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f670c0b3cd0 0x7f670c153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 --2- 172.21.15.28:0/3286195330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f670c06bc50 0x7f670c141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 >> 172.21.15.28:0/3286195330 conn(0x7f670c0710b0 msgr2=0x7f670c06a530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 shutdown_connections 2023-12-16T21:58:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.655+0000 7f6713393640 1 -- 172.21.15.28:0/3286195330 wait complete. 2023-12-16T21:58:08.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:08.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe850af640 1 Processor -- start 2023-12-16T21:58:08.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe850af640 1 -- start start 2023-12-16T21:58:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe850af640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe800d1780 0x7efe800d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe850af640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efe800cfac0 0x7efe800cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe850af640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe800d0d30 con 0x7efe800b3bc0 2023-12-16T21:58:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe850af640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe800d0490 con 0x7efe800d1780 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe850af640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe80160f80 con 0x7efe800cfac0 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe7e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe800d1780 0x7efe800d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe7ed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efe800b3bc0 0x7efe800b2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43828/0 (socket says 172.21.15.28:43828) 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe7e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe800d1780 0x7efe800d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60200/0 (socket says 172.21.15.28:60200) 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.791+0000 7efe7ed9d640 1 -- 172.21.15.28:0/2726415518 learned_addr learned my addr 172.21.15.28:0/2726415518 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7f59e640 1 --2- 172.21.15.28:0/2726415518 >> v2:172.21.15.122:3300/0 conn(0x7efe800cfac0 0x7efe800cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7e59c640 1 -- 172.21.15.28:0/2726415518 >> v2:172.21.15.122:3300/0 conn(0x7efe800cfac0 msgr2=0x7efe800cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7e59c640 1 --2- 172.21.15.28:0/2726415518 >> v2:172.21.15.122:3300/0 conn(0x7efe800cfac0 0x7efe800cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7e59c640 1 -- 172.21.15.28:0/2726415518 >> v1:172.21.15.28:6789/0 conn(0x7efe800b3bc0 legacy=0x7efe800b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7e59c640 1 -- 172.21.15.28:0/2726415518 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe800d12f0 con 0x7efe800d1780 2023-12-16T21:58:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7dd9b640 1 -- 172.21.15.28:0/2726415518 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543965517 0 0) 0x7efe800d0d30 con 0x7efe800b3bc0 2023-12-16T21:58:08.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7e59c640 1 --2- 172.21.15.28:0/2726415518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe800d1be0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efe800b4bc0 tx=0x7efe7401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f887c55b5312e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7dd9b640 1 -- 172.21.15.28:0/2726415518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe740ab600 con 0x7efe800d1780 2023-12-16T21:58:08.798 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7dd9b640 1 -- 172.21.15.28:0/2726415518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe740a7050 con 0x7efe800d1780 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe7dd9b640 1 -- 172.21.15.28:0/2726415518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe740ab980 con 0x7efe800d1780 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe850af640 1 -- 172.21.15.28:0/2726415518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 msgr2=0x7efe800d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe850af640 1 --2- 172.21.15.28:0/2726415518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe800d1be0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7efe800b4bc0 tx=0x7efe7401c440 comp rx=0 tx=0).stop 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe850af640 1 -- 172.21.15.28:0/2726415518 shutdown_connections 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe850af640 1 --2- 172.21.15.28:0/2726415518 >> v2:172.21.15.122:3300/0 conn(0x7efe800cfac0 0x7efe800cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe850af640 1 --2- 172.21.15.28:0/2726415518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe800d1be0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe850af640 1 -- 172.21.15.28:0/2726415518 >> 172.21.15.28:0/2726415518 conn(0x7efe8009c430 msgr2=0x7efe800d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe850af640 1 -- 172.21.15.28:0/2726415518 shutdown_connections 2023-12-16T21:58:08.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.795+0000 7efe850af640 1 -- 172.21.15.28:0/2726415518 wait complete. 2023-12-16T21:58:08.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 Processor -- start 2023-12-16T21:58:08.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 -- start start 2023-12-16T21:58:08.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efe800b3bc0 0x7efe8009f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe800cfac0 0x7efe8015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe8014e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8015de20 con 0x7efe800d1780 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8015d840 con 0x7efe800cfac0 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8015cc30 con 0x7efe800b3bc0 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe8014e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe8014e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60202/0 (socket says 172.21.15.28:60202) 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 -- 172.21.15.28:0/1870642590 learned_addr learned my addr 172.21.15.28:0/1870642590 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7ed9d640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efe800b3bc0 0x7efe8009f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7e59c640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe800cfac0 0x7efe8015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 -- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efe800b3bc0 msgr2=0x7efe8009f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efe800b3bc0 0x7efe8009f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 -- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe800cfac0 msgr2=0x7efe8015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe800cfac0 0x7efe8015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 -- 172.21.15.28:0/1870642590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe74009420 con 0x7efe800d1780 2023-12-16T21:58:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe7f59e640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe8014e470 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efe78007e60 tx=0x7efe7800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe677fe640 1 -- 172.21.15.28:0/1870642590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78014420 con 0x7efe800d1780 2023-12-16T21:58:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe677fe640 1 -- 172.21.15.28:0/1870642590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe780145a0 con 0x7efe800d1780 2023-12-16T21:58:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe677fe640 1 -- 172.21.15.28:0/1870642590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe78014950 con 0x7efe800d1780 2023-12-16T21:58:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe8015cf10 con 0x7efe800d1780 2023-12-16T21:58:08.803 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8015d4d0 con 0x7efe800d1780 2023-12-16T21:58:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.799+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe800674d0 con 0x7efe800d1780 2023-12-16T21:58:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.803+0000 7efe677fe640 1 -- 172.21.15.28:0/1870642590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7efe78014ab0 con 0x7efe800d1780 2023-12-16T21:58:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.803+0000 7efe677fe640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efe6c066470 0x7efe6c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.803+0000 7efe677fe640 1 -- 172.21.15.28:0/1870642590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe78084770 con 0x7efe800d1780 2023-12-16T21:58:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.803+0000 7efe7ed9d640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efe6c066470 0x7efe6c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:08.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.803+0000 7efe7ed9d640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efe6c066470 0x7efe6c068930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efe70000f70 tx=0x7efe70007c10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:08.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.807+0000 7efe677fe640 1 -- 172.21.15.28:0/1870642590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7efe78051c20 con 0x7efe800d1780 2023-12-16T21:58:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.963+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe800b26c0 con 0x7efe800d1780 2023-12-16T21:58:08.967 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe677fe640 1 -- 172.21.15.28:0/1870642590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7efe78055240 con 0x7efe800d1780 2023-12-16T21:58:08.967 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:08.968 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efe6c066470 msgr2=0x7efe6c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efe6c066470 0x7efe6c068930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efe70000f70 tx=0x7efe70007c10 comp rx=0 tx=0).stop 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 msgr2=0x7efe8014e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe8014e470 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7efe78007e60 tx=0x7efe7800e4b0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 shutdown_connections 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efe6c066470 0x7efe6c068930 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe800d1780 0x7efe8014e470 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe800cfac0 0x7efe8015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 --2- 172.21.15.28:0/1870642590 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efe800b3bc0 0x7efe8009f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 >> 172.21.15.28:0/1870642590 conn(0x7efe8009c430 msgr2=0x7efe800ccc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 shutdown_connections 2023-12-16T21:58:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:08.967+0000 7efe850af640 1 -- 172.21.15.28:0/1870642590 wait complete. 2023-12-16T21:58:08.986 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:08.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8ac63e640 1 Processor -- start 2023-12-16T21:58:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8ac63e640 1 -- start start 2023-12-16T21:58:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8ac63e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 0x7fe8a4160de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8ac63e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe8a40ac840 0x7fe8a4161320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8ac63e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8a4146a60 0x7fe8a4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8ac63e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a40b2390 con 0x7fe8a4146a60 2023-12-16T21:58:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8ac63e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a4146510 con 0x7fe8a40ac840 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8ac63e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40b2060 con 0x7fe8a4145ea0 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aabdb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8a4146a60 0x7fe8a4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aabdb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8a4146a60 0x7fe8a4070e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60216/0 (socket says 172.21.15.28:60216) 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aabdb640 1 -- 172.21.15.28:0/3101618159 learned_addr learned my addr 172.21.15.28:0/3101618159 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8a9bd9640 1 --2- 172.21.15.28:0/3101618159 >> v2:172.21.15.122:3300/0 conn(0x7fe8a40ac840 0x7fe8a4161320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aa3da640 1 --2- 172.21.15.28:0/3101618159 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 0x7fe8a4160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aabdb640 1 -- 172.21.15.28:0/3101618159 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 msgr2=0x7fe8a4160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aabdb640 1 --2- 172.21.15.28:0/3101618159 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 0x7fe8a4160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aabdb640 1 -- 172.21.15.28:0/3101618159 >> v2:172.21.15.122:3300/0 conn(0x7fe8a40ac840 msgr2=0x7fe8a4161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aabdb640 1 --2- 172.21.15.28:0/3101618159 >> v2:172.21.15.122:3300/0 conn(0x7fe8a40ac840 0x7fe8a4161320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.115+0000 7fe8aabdb640 1 -- 172.21.15.28:0/3101618159 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a41467b0 con 0x7fe8a4146a60 2023-12-16T21:58:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8aabdb640 1 --2- 172.21.15.28:0/3101618159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4070e00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe8a406c080 tx=0x7fe89c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b652d6b817d50165 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8a93d8640 1 -- 172.21.15.28:0/3101618159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c0a5700 con 0x7fe8a4146a60 2023-12-16T21:58:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8a93d8640 1 -- 172.21.15.28:0/3101618159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c0a5860 con 0x7fe8a4146a60 2023-12-16T21:58:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8a93d8640 1 -- 172.21.15.28:0/3101618159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c0a5bf0 con 0x7fe8a4146a60 2023-12-16T21:58:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- 172.21.15.28:0/3101618159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 msgr2=0x7fe8a4070e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/3101618159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4070e00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe8a406c080 tx=0x7fe89c01c9d0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- 172.21.15.28:0/3101618159 shutdown_connections 2023-12-16T21:58:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/3101618159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4070e00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/3101618159 >> v2:172.21.15.122:3300/0 conn(0x7fe8a40ac840 0x7fe8a4161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/3101618159 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 0x7fe8a4160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- 172.21.15.28:0/3101618159 >> 172.21.15.28:0/3101618159 conn(0x7fe8a40b0cc0 msgr2=0x7fe8a40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- 172.21.15.28:0/3101618159 shutdown_connections 2023-12-16T21:58:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- 172.21.15.28:0/3101618159 wait complete. 2023-12-16T21:58:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 Processor -- start 2023-12-16T21:58:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- start start 2023-12-16T21:58:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a40ac840 0x7fe8a415a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 0x7fe8a4159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a4136d40 con 0x7fe8a4146a60 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a415e6c0 con 0x7fe8a40ac840 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8ac63e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a415e9a0 con 0x7fe8a4145ea0 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8aabdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8aabdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4159ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60222/0 (socket says 172.21.15.28:60222) 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8aabdb640 1 -- 172.21.15.28:0/145864709 learned_addr learned my addr 172.21.15.28:0/145864709 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8aa3da640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a40ac840 0x7fe8a415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8a9bd9640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 0x7fe8a4159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8aabdb640 1 -- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 msgr2=0x7fe8a4159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8aabdb640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 0x7fe8a4159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.119+0000 7fe8aabdb640 1 -- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a40ac840 msgr2=0x7fe8a415a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe8aabdb640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a40ac840 0x7fe8a415a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe8aabdb640 1 -- 172.21.15.28:0/145864709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c009b30 con 0x7fe8a4146a60 2023-12-16T21:58:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe8aabdb640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4159ad0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0af040 tx=0x7fe89c019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe896ffd640 1 -- 172.21.15.28:0/145864709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c0b8640 con 0x7fe8a4146a60 2023-12-16T21:58:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe896ffd640 1 -- 172.21.15.28:0/145864709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c0b87a0 con 0x7fe8a4146a60 2023-12-16T21:58:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a415ec80 con 0x7fe8a4146a60 2023-12-16T21:58:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe896ffd640 1 -- 172.21.15.28:0/145864709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c0b8b10 con 0x7fe8a4146a60 2023-12-16T21:58:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a415e230 con 0x7fe8a4146a60 2023-12-16T21:58:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a415e420 con 0x7fe8a4146a60 2023-12-16T21:58:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe896ffd640 1 -- 172.21.15.28:0/145864709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe89c0b8c70 con 0x7fe8a4146a60 2023-12-16T21:58:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe896ffd640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe8880664a0 0x7fe888068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe896ffd640 1 -- 172.21.15.28:0/145864709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe89c11edb0 con 0x7fe8a4146a60 2023-12-16T21:58:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe8aa3da640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe8880664a0 0x7fe888068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.123+0000 7fe8aa3da640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe8880664a0 0x7fe888068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4168140 tx=0x7fe8a0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.127+0000 7fe896ffd640 1 -- 172.21.15.28:0/145864709 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe89c0ec3e0 con 0x7fe8a4146a60 2023-12-16T21:58:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.283+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8a40693d0 con 0x7fe8a4146a60 2023-12-16T21:58:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.283+0000 7fe896ffd640 1 -- 172.21.15.28:0/145864709 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe89c0efa00 con 0x7fe8a4146a60 2023-12-16T21:58:09.286 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe8880664a0 msgr2=0x7fe888068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe8880664a0 0x7fe888068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4168140 tx=0x7fe8a0008040 comp rx=0 tx=0).stop 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 msgr2=0x7fe8a4159ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4159ad0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0af040 tx=0x7fe89c019770 comp rx=0 tx=0).stop 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 shutdown_connections 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe8880664a0 0x7fe888068960 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8a4146a60 0x7fe8a4159ad0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe8a4145ea0 0x7fe8a4159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 --2- 172.21.15.28:0/145864709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8a40ac840 0x7fe8a415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 >> 172.21.15.28:0/145864709 conn(0x7fe8a40b0cc0 msgr2=0x7fe8a4160d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 shutdown_connections 2023-12-16T21:58:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.287+0000 7fe8ac63e640 1 -- 172.21.15.28:0/145864709 wait complete. 2023-12-16T21:58:09.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f547e4e7640 1 Processor -- start 2023-12-16T21:58:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f547e4e7640 1 -- start start 2023-12-16T21:58:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f547e4e7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5478071590 0x7f5478071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f547e4e7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547806ff20 con 0x7f54780b2fd0 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f547e4e7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5478072a70 con 0x7f5478143b30 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f547e4e7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5478071f50 con 0x7f5478071590 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f54777fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5478071590 0x7f5478071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f547ca84640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f54780b2fd0 0x7f54780b3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43840/0 (socket says 172.21.15.28:43840) 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f547ca84640 1 -- 172.21.15.28:0/3752037447 learned_addr learned my addr 172.21.15.28:0/3752037447 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f54777fe640 1 -- 172.21.15.28:0/3752037447 >> v1:172.21.15.122:6789/0 conn(0x7f5478143b30 legacy=0x7f5478072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f54777fe640 1 -- 172.21.15.28:0/3752037447 >> v1:172.21.15.28:6789/0 conn(0x7f54780b2fd0 legacy=0x7f54780b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.427+0000 7f54777fe640 1 -- 172.21.15.28:0/3752037447 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54780cec90 con 0x7f5478071590 2023-12-16T21:58:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f54777fe640 1 --2- 172.21.15.28:0/3752037447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478071a10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f546c0097c0 tx=0x7f546c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec88bd48587d265 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f5476ffd640 1 -- 172.21.15.28:0/3752037447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c0ad680 con 0x7f5478071590 2023-12-16T21:58:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f5476ffd640 1 -- 172.21.15.28:0/3752037447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c0ad7e0 con 0x7f5478071590 2023-12-16T21:58:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f5476ffd640 1 -- 172.21.15.28:0/3752037447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c0adb70 con 0x7f5478071590 2023-12-16T21:58:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 -- 172.21.15.28:0/3752037447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 msgr2=0x7f5478071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 --2- 172.21.15.28:0/3752037447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478071a10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f546c0097c0 tx=0x7f546c01c380 comp rx=0 tx=0).stop 2023-12-16T21:58:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 -- 172.21.15.28:0/3752037447 shutdown_connections 2023-12-16T21:58:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 --2- 172.21.15.28:0/3752037447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478071a10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 -- 172.21.15.28:0/3752037447 >> 172.21.15.28:0/3752037447 conn(0x7f547806d510 msgr2=0x7f547806d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 -- 172.21.15.28:0/3752037447 shutdown_connections 2023-12-16T21:58:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 -- 172.21.15.28:0/3752037447 wait complete. 2023-12-16T21:58:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 Processor -- start 2023-12-16T21:58:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.431+0000 7f547e4e7640 1 -- start start 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54780b2fd0 0x7f547806a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5478143b30 0x7f547806aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547813beb0 con 0x7f5478071590 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5478159880 con 0x7f54780b2fd0 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5478159da0 con 0x7f5478143b30 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60242/0 (socket says 172.21.15.28:60242) 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 -- 172.21.15.28:0/1490751103 learned_addr learned my addr 172.21.15.28:0/1490751103 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f54777fe640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54780b2fd0 0x7f547806a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547ca84640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5478143b30 0x7f547806aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 -- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5478143b30 msgr2=0x7f547806aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5478143b30 0x7f547806aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 -- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54780b2fd0 msgr2=0x7f547806a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54780b2fd0 0x7f547806a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 -- 172.21.15.28:0/1490751103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c009420 con 0x7f5478071590 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5477fff640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478159340 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f547000c7c0 tx=0x7f547000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5474ff9640 1 -- 172.21.15.28:0/1490751103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547000aa50 con 0x7f5478071590 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5474ff9640 1 -- 172.21.15.28:0/1490751103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547000abb0 con 0x7f5478071590 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547815a740 con 0x7f5478071590 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5474ff9640 1 -- 172.21.15.28:0/1490751103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470012420 con 0x7f5478071590 2023-12-16T21:58:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547815ad00 con 0x7f5478071590 2023-12-16T21:58:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54780674d0 con 0x7f5478071590 2023-12-16T21:58:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5474ff9640 1 -- 172.21.15.28:0/1490751103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f5470012580 con 0x7f5478071590 2023-12-16T21:58:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5474ff9640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5464066470 0x7f5464068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f5474ff9640 1 -- 172.21.15.28:0/1490751103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5470084160 con 0x7f5478071590 2023-12-16T21:58:09.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.435+0000 7f54777fe640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5464066470 0x7f5464068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.439+0000 7f54777fe640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5464066470 0x7f5464068930 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f546c0af000 tx=0x7f546c0a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.439+0000 7f5474ff9640 1 -- 172.21.15.28:0/1490751103 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f5470051550 con 0x7f5478071590 2023-12-16T21:58:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f547806fa60 con 0x7f5478071590 2023-12-16T21:58:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f5474ff9640 1 -- 172.21.15.28:0/1490751103 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5470054b70 con 0x7f5478071590 2023-12-16T21:58:09.596 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5464066470 msgr2=0x7f5464068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5464066470 0x7f5464068930 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f546c0af000 tx=0x7f546c0a7000 comp rx=0 tx=0).stop 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 msgr2=0x7f5478159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478159340 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f547000c7c0 tx=0x7f547000cc80 comp rx=0 tx=0).stop 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 shutdown_connections 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5464066470 0x7f5464068930 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5478143b30 0x7f547806aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f54780b2fd0 0x7f547806a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 --2- 172.21.15.28:0/1490751103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5478071590 0x7f5478159340 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 >> 172.21.15.28:0/1490751103 conn(0x7f547806d510 msgr2=0x7f547806d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 shutdown_connections 2023-12-16T21:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.595+0000 7f547e4e7640 1 -- 172.21.15.28:0/1490751103 wait complete. 2023-12-16T21:58:09.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:09.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f858cf96640 1 Processor -- start 2023-12-16T21:58:09.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f858cf96640 1 -- start start 2023-12-16T21:58:09.752 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f858cf96640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f85881571d0 0x7f8588070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f858cf96640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f858815d8e0 0x7f85880ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f858cf96640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85881575b0 con 0x7f85880ae830 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f858cf96640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8588071600 con 0x7f85881571d0 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f858cf96640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85880718e0 con 0x7f858815d8e0 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f8585d9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f85880ae830 0x7f858815d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43848/0 (socket says 172.21.15.28:43848) 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.747+0000 7f8585d9b640 1 -- 172.21.15.28:0/321434667 learned_addr learned my addr 172.21.15.28:0/321434667 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858659c640 1 --2- 172.21.15.28:0/321434667 >> v2:172.21.15.122:3300/0 conn(0x7f85881571d0 0x7f8588070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f858815d8e0 0x7f85880ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448433948 0 0) 0x7f85881575b0 con 0x7f85880ae830 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8574003530 con 0x7f85880ae830 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3614285831 0 0) 0x7f8574003530 con 0x7f85880ae830 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f858815d8e0 msgr2=0x7f85880ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f858815d8e0 0x7f85880ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 >> v2:172.21.15.122:3300/0 conn(0x7f85881571d0 msgr2=0x7f8588070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 --2- 172.21.15.28:0/321434667 >> v2:172.21.15.122:3300/0 conn(0x7f85881571d0 0x7f8588070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588071bc0 con 0x7f85880ae830 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f85700016a0 con 0x7f85880ae830 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 >> v1:172.21.15.28:6789/0 conn(0x7f85880ae830 legacy=0x7f858815d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8574003980 0x7f8574003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f858815d8e0 0x7f85740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8574004d10 0x7f8574005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8588071600 con 0x7f8574004d10 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85880718e0 con 0x7f8574003980 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857400b990 con 0x7f858815d8e0 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8574004d10 0x7f8574005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858659c640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8574003980 0x7f8574003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8585d9b640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f858815d8e0 0x7f85740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 -- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f858815d8e0 msgr2=0x7f85740047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f858815d8e0 0x7f85740047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 -- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8574003980 msgr2=0x7f8574003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8574003980 0x7f8574003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 -- 172.21.15.28:0/321434667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8574003530 con 0x7f8574004d10 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8574004d10 0x7f8574005190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8580009ab0 tx=0x7f858001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388352ba094bd508 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85800a55e0 con 0x7f8574004d10 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85800a5740 con 0x7f8574004d10 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858559a640 1 -- 172.21.15.28:0/321434667 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85800a5ab0 con 0x7f8574004d10 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858cf96640 1 -- 172.21.15.28:0/321434667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8574004d10 msgr2=0x7f8574005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858cf96640 1 --2- 172.21.15.28:0/321434667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8574004d10 0x7f8574005190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8580009ab0 tx=0x7f858001c9f0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f8586d9d640 1 -- 172.21.15.28:0/321434667 reap_dead start 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858cf96640 1 -- 172.21.15.28:0/321434667 shutdown_connections 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.751+0000 7f858cf96640 1 -- 172.21.15.28:0/321434667 >> 172.21.15.28:0/321434667 conn(0x7f85880b0cc0 msgr2=0x7f85880b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- 172.21.15.28:0/321434667 shutdown_connections 2023-12-16T21:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- 172.21.15.28:0/321434667 wait complete. 2023-12-16T21:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 Processor -- start 2023-12-16T21:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- start start 2023-12-16T21:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f85880ce2a0 0x7f85880b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880ce680 0x7f85880b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f858813ccd0 0x7f858813a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85880b2dc0 con 0x7f85880ce680 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85880b30a0 con 0x7f858813ccd0 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85880b3380 con 0x7f85880ce2a0 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880ce680 0x7f85880b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880ce680 0x7f85880b3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60270/0 (socket says 172.21.15.28:60270) 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 -- 172.21.15.28:0/693079554 learned_addr learned my addr 172.21.15.28:0/693079554 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858659c640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f85880ce2a0 0x7f85880b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8586d9d640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f858813ccd0 0x7f858813a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 -- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f85880ce2a0 msgr2=0x7f85880b27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f85880ce2a0 0x7f85880b27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 -- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f858813ccd0 msgr2=0x7f858813a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f858813ccd0 0x7f858813a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 -- 172.21.15.28:0/693079554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85880b3660 con 0x7f85880ce680 2023-12-16T21:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f8585d9b640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880ce680 0x7f85880b3e40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8570001ec0 tx=0x7f857000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f857effd640 1 -- 172.21.15.28:0/693079554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857000c8d0 con 0x7f85880ce680 2023-12-16T21:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f857effd640 1 -- 172.21.15.28:0/693079554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857000ca50 con 0x7f85880ce680 2023-12-16T21:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85880b3940 con 0x7f85880ce680 2023-12-16T21:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f857effd640 1 -- 172.21.15.28:0/693079554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857000ce00 con 0x7f85880ce680 2023-12-16T21:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858813a750 con 0x7f85880ce680 2023-12-16T21:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.755+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85880693d0 con 0x7f85880ce680 2023-12-16T21:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.759+0000 7f857effd640 1 -- 172.21.15.28:0/693079554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f857001d460 con 0x7f85880ce680 2023-12-16T21:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.759+0000 7f857effd640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8568066470 0x7f8568068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.759+0000 7f857effd640 1 -- 172.21.15.28:0/693079554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8570083fa0 con 0x7f85880ce680 2023-12-16T21:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.759+0000 7f858659c640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8568066470 0x7f8568068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.759+0000 7f858659c640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8568066470 0x7f8568068930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f857800a240 tx=0x7f857800c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.763+0000 7f857effd640 1 -- 172.21.15.28:0/693079554 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f85700b5290 con 0x7f85880ce680 2023-12-16T21:58:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.911+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f85881587a0 con 0x7f85880ce680 2023-12-16T21:58:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.911+0000 7f857effd640 1 -- 172.21.15.28:0/693079554 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8570051420 con 0x7f85880ce680 2023-12-16T21:58:09.915 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8568066470 msgr2=0x7f8568068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8568066470 0x7f8568068930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f857800a240 tx=0x7f857800c040 comp rx=0 tx=0).stop 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880ce680 msgr2=0x7f85880b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880ce680 0x7f85880b3e40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8570001ec0 tx=0x7f857000ed30 comp rx=0 tx=0).stop 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 shutdown_connections 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8568066470 0x7f8568068930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f858813ccd0 0x7f858813a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880ce680 0x7f85880b3e40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 --2- 172.21.15.28:0/693079554 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f85880ce2a0 0x7f85880b27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 >> 172.21.15.28:0/693079554 conn(0x7f85880b0cc0 msgr2=0x7f85880af440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 shutdown_connections 2023-12-16T21:58:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:09.915+0000 7f858cf96640 1 -- 172.21.15.28:0/693079554 wait complete. 2023-12-16T21:58:09.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb2491f640 1 Processor -- start 2023-12-16T21:58:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb2491f640 1 -- start start 2023-12-16T21:58:10.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb2491f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb20160870 0x7fcb200d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb2491f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcb200cdb20 0x7fcb200b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb2491f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb200d0d70 con 0x7fcb200b51a0 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb2491f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb20160c50 con 0x7fcb200cdb20 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb2491f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb200ce2e0 con 0x7fcb20160870 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1ed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcb200b51a0 0x7fcb200d17c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43856/0 (socket says 172.21.15.28:43856) 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1ed9d640 1 -- 172.21.15.28:0/1316285750 learned_addr learned my addr 172.21.15.28:0/1316285750 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1dd9b640 1 --2- 172.21.15.28:0/1316285750 >> v2:172.21.15.28:3300/0 conn(0x7fcb200cdb20 0x7fcb200b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1e59c640 1 --2- 172.21.15.28:0/1316285750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb20160870 0x7fcb200d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1d59a640 1 -- 172.21.15.28:0/1316285750 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656163569 0 0) 0x7fcb200d0d70 con 0x7fcb200b51a0 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1d59a640 1 -- 172.21.15.28:0/1316285750 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb08003530 con 0x7fcb200b51a0 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1dd9b640 1 -- 172.21.15.28:0/1316285750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb20160870 msgr2=0x7fcb200d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1dd9b640 1 --2- 172.21.15.28:0/1316285750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb20160870 0x7fcb200d2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1dd9b640 1 -- 172.21.15.28:0/1316285750 >> v1:172.21.15.28:6789/0 conn(0x7fcb200b51a0 legacy=0x7fcb200d17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.043+0000 7fcb1dd9b640 1 -- 172.21.15.28:0/1316285750 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200ce5c0 con 0x7fcb200cdb20 2023-12-16T21:58:10.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb1dd9b640 1 --2- 172.21.15.28:0/1316285750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200b6230 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcb140097c0 tx=0x7fcb1401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d7fdadc140ce63 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb1d59a640 1 -- 172.21.15.28:0/1316285750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb140ad640 con 0x7fcb200cdb20 2023-12-16T21:58:10.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb1d59a640 1 -- 172.21.15.28:0/1316285750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb140ad7a0 con 0x7fcb200cdb20 2023-12-16T21:58:10.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb1d59a640 1 -- 172.21.15.28:0/1316285750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb140adb30 con 0x7fcb200cdb20 2023-12-16T21:58:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- 172.21.15.28:0/1316285750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 msgr2=0x7fcb200b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 --2- 172.21.15.28:0/1316285750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200b6230 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcb140097c0 tx=0x7fcb1401c480 comp rx=0 tx=0).stop 2023-12-16T21:58:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- 172.21.15.28:0/1316285750 shutdown_connections 2023-12-16T21:58:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 --2- 172.21.15.28:0/1316285750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200b6230 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 --2- 172.21.15.28:0/1316285750 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb20160870 0x7fcb200d2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- 172.21.15.28:0/1316285750 >> 172.21.15.28:0/1316285750 conn(0x7fcb20072060 msgr2=0x7fcb20072470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- 172.21.15.28:0/1316285750 shutdown_connections 2023-12-16T21:58:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- 172.21.15.28:0/1316285750 wait complete. 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 Processor -- start 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- start start 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb200b51a0 0x7fcb2014c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb20160870 0x7fcb20158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2014c6b0 con 0x7fcb200cdb20 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2014db60 con 0x7fcb20160870 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb2491f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2015b5d0 con 0x7fcb200b51a0 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.047+0000 7fcb1dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200c93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60286/0 (socket says 172.21.15.28:60286) 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1dd9b640 1 -- 172.21.15.28:0/3641808051 learned_addr learned my addr 172.21.15.28:0/3641808051 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1ed9d640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb20160870 0x7fcb20158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1e59c640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb200b51a0 0x7fcb2014c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1dd9b640 1 -- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb200b51a0 msgr2=0x7fcb2014c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1dd9b640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb200b51a0 0x7fcb2014c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1dd9b640 1 -- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb20160870 msgr2=0x7fcb20158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1dd9b640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb20160870 0x7fcb20158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1dd9b640 1 -- 172.21.15.28:0/3641808051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb14009420 con 0x7fcb200cdb20 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1e59c640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb200b51a0 0x7fcb2014c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1dd9b640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200c93d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcb1401a0b0 tx=0x7fcb14019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb0effd640 1 -- 172.21.15.28:0/3641808051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb140b6610 con 0x7fcb200cdb20 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb0effd640 1 -- 172.21.15.28:0/3641808051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb140a6040 con 0x7fcb200cdb20 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb200b5b40 con 0x7fcb200cdb20 2023-12-16T21:58:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb0effd640 1 -- 172.21.15.28:0/3641808051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb140b69d0 con 0x7fcb200cdb20 2023-12-16T21:58:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb200c8610 con 0x7fcb200cdb20 2023-12-16T21:58:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb20069440 con 0x7fcb200cdb20 2023-12-16T21:58:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb0effd640 1 -- 172.21.15.28:0/3641808051 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fcb140b4040 con 0x7fcb200cdb20 2023-12-16T21:58:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb0effd640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcb000664a0 0x7fcb00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb0effd640 1 -- 172.21.15.28:0/3641808051 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb1411dea0 con 0x7fcb200cdb20 2023-12-16T21:58:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1e59c640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcb000664a0 0x7fcb00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.051+0000 7fcb1e59c640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcb000664a0 0x7fcb00068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009f50 tx=0x7fcb1000b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.055+0000 7fcb0effd640 1 -- 172.21.15.28:0/3641808051 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fcb140eb320 con 0x7fcb200cdb20 2023-12-16T21:58:10.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.203+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb200b62a0 con 0x7fcb200cdb20 2023-12-16T21:58:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb0effd640 1 -- 172.21.15.28:0/3641808051 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fcb140b2090 con 0x7fcb200cdb20 2023-12-16T21:58:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:10.208 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcb000664a0 msgr2=0x7fcb00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcb000664a0 0x7fcb00068960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcb10009f50 tx=0x7fcb1000b040 comp rx=0 tx=0).stop 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 msgr2=0x7fcb200c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200c93d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcb1401a0b0 tx=0x7fcb14019220 comp rx=0 tx=0).stop 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 shutdown_connections 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcb000664a0 0x7fcb00068960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb20160870 0x7fcb20158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb200cdb20 0x7fcb200c93d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 --2- 172.21.15.28:0/3641808051 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcb200b51a0 0x7fcb2014c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 >> 172.21.15.28:0/3641808051 conn(0x7fcb20072060 msgr2=0x7fcb200b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:10.211 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 shutdown_connections 2023-12-16T21:58:10.212 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.207+0000 7fcb2491f640 1 -- 172.21.15.28:0/3641808051 wait complete. 2023-12-16T21:58:10.226 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:10.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.347+0000 7ffb7b995640 1 Processor -- start 2023-12-16T21:58:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.347+0000 7ffb7b995640 1 -- start start 2023-12-16T21:58:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb7b995640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffb7415adb0 0x7ffb7413d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb7b995640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb7413f5f0 0x7ffb740ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb7b995640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7413d2b0 con 0x7ffb74071d90 2023-12-16T21:58:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb7b995640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb74140520 con 0x7ffb7415adb0 2023-12-16T21:58:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb7b995640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740cd7f0 con 0x7ffb7413f5f0 2023-12-16T21:58:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb78f30640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffb7415adb0 0x7ffb7413d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb78f30640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffb7415adb0 0x7ffb7413d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60290/0 (socket says 172.21.15.28:60290) 2023-12-16T21:58:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb78f30640 1 -- 172.21.15.28:0/434255461 learned_addr learned my addr 172.21.15.28:0/434255461 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb78f30640 1 -- 172.21.15.28:0/434255461 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb7413f5f0 msgr2=0x7ffb740ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb78f30640 1 --2- 172.21.15.28:0/434255461 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb7413f5f0 0x7ffb740ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb78f30640 1 -- 172.21.15.28:0/434255461 >> v1:172.21.15.122:6789/0 conn(0x7ffb74071d90 legacy=0x7ffb74072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb78f30640 1 -- 172.21.15.28:0/434255461 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb740cdd70 con 0x7ffb7415adb0 2023-12-16T21:58:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb78f30640 1 --2- 172.21.15.28:0/434255461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb7413d560 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009870 tx=0x7ffb6401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b497ffe7fbe6777f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb6bfff640 1 -- 172.21.15.28:0/434255461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640aa5f0 con 0x7ffb7415adb0 2023-12-16T21:58:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb6bfff640 1 -- 172.21.15.28:0/434255461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb640aa750 con 0x7ffb7415adb0 2023-12-16T21:58:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb6bfff640 1 -- 172.21.15.28:0/434255461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640aaae0 con 0x7ffb7415adb0 2023-12-16T21:58:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb7b995640 1 -- 172.21.15.28:0/434255461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 msgr2=0x7ffb7413d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.351+0000 7ffb7b995640 1 --2- 172.21.15.28:0/434255461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb7413d560 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009870 tx=0x7ffb6401c4d0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- 172.21.15.28:0/434255461 shutdown_connections 2023-12-16T21:58:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 --2- 172.21.15.28:0/434255461 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb7413f5f0 0x7ffb740ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 --2- 172.21.15.28:0/434255461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb7413d560 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- 172.21.15.28:0/434255461 >> 172.21.15.28:0/434255461 conn(0x7ffb740b0cc0 msgr2=0x7ffb740b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- 172.21.15.28:0/434255461 shutdown_connections 2023-12-16T21:58:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- 172.21.15.28:0/434255461 wait complete. 2023-12-16T21:58:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 Processor -- start 2023-12-16T21:58:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- start start 2023-12-16T21:58:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb74071d90 0x7ffb74134ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb7413f5f0 0x7ffb74143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb74143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740b2e80 con 0x7ffb7415adb0 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb74151590 con 0x7ffb7413f5f0 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb74152570 con 0x7ffb74071d90 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb74143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb74143590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60298/0 (socket says 172.21.15.28:60298) 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 -- 172.21.15.28:0/1601660046 learned_addr learned my addr 172.21.15.28:0/1601660046 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb78f30640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb7413f5f0 0x7ffb74143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 -- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb74071d90 msgr2=0x7ffb74134ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79731640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb74071d90 0x7ffb74134ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb74071d90 0x7ffb74134ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 -- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb7413f5f0 msgr2=0x7ffb74143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb7413f5f0 0x7ffb74143050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 -- 172.21.15.28:0/1601660046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb640094d0 con 0x7ffb7415adb0 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb78f30640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb7413f5f0 0x7ffb74143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb79f32640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb74143590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000b9a0 tx=0x7ffb7000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb69ffb640 1 -- 172.21.15.28:0/1601660046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7000cba0 con 0x7ffb7415adb0 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb69ffb640 1 -- 172.21.15.28:0/1601660046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7000cd20 con 0x7ffb7415adb0 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb74151fb0 con 0x7ffb7415adb0 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb69ffb640 1 -- 172.21.15.28:0/1601660046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70014680 con 0x7ffb7415adb0 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb7413b980 con 0x7ffb7415adb0 2023-12-16T21:58:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.355+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb740b2fc0 con 0x7ffb7415adb0 2023-12-16T21:58:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.359+0000 7ffb69ffb640 1 -- 172.21.15.28:0/1601660046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7ffb7001d3b0 con 0x7ffb7415adb0 2023-12-16T21:58:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.359+0000 7ffb69ffb640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffb58066470 0x7ffb58068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.359+0000 7ffb69ffb640 1 -- 172.21.15.28:0/1601660046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb70083e30 con 0x7ffb7415adb0 2023-12-16T21:58:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.359+0000 7ffb79731640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffb58066470 0x7ffb58068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.359+0000 7ffb79731640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffb58066470 0x7ffb58068930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00a790 tx=0x7ffb6c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.363+0000 7ffb69ffb640 1 -- 172.21.15.28:0/1601660046 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7ffb740b2fc0 con 0x7ffb7415adb0 2023-12-16T21:58:10.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb740b2fc0 con 0x7ffb7415adb0 2023-12-16T21:58:10.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb69ffb640 1 -- 172.21.15.28:0/1601660046 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb740b2fc0 con 0x7ffb7415adb0 2023-12-16T21:58:10.520 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffb58066470 msgr2=0x7ffb58068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffb58066470 0x7ffb58068930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c00a790 tx=0x7ffb6c008040 comp rx=0 tx=0).stop 2023-12-16T21:58:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 msgr2=0x7ffb74143590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb74143590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000b9a0 tx=0x7ffb7000be60 comp rx=0 tx=0).stop 2023-12-16T21:58:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 shutdown_connections 2023-12-16T21:58:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffb58066470 0x7ffb58068930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb7415adb0 0x7ffb74143590 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb7413f5f0 0x7ffb74143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 --2- 172.21.15.28:0/1601660046 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb74071d90 0x7ffb74134ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 >> 172.21.15.28:0/1601660046 conn(0x7ffb740b0cc0 msgr2=0x7ffb740af830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:10.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 shutdown_connections 2023-12-16T21:58:10.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.519+0000 7ffb7b995640 1 -- 172.21.15.28:0/1601660046 wait complete. 2023-12-16T21:58:10.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.659+0000 7f39693e9640 1 Processor -- start 2023-12-16T21:58:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.659+0000 7f39693e9640 1 -- start start 2023-12-16T21:58:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.659+0000 7f39693e9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f396415adb0 0x7f396413d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.659+0000 7f39693e9640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396413d2b0 con 0x7f396413f5f0 2023-12-16T21:58:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.659+0000 7f39693e9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640cdd70 con 0x7f3964071d90 2023-12-16T21:58:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.659+0000 7f39693e9640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3964140520 con 0x7f396415adb0 2023-12-16T21:58:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.659+0000 7f39627fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f396415adb0 0x7f396413d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39627fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f396415adb0 0x7f396413d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60310/0 (socket says 172.21.15.28:60310) 2023-12-16T21:58:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39637fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f396413f5f0 0x7f39640ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43864/0 (socket says 172.21.15.28:43864) 2023-12-16T21:58:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39627fc640 1 -- 172.21.15.28:0/2568354 learned_addr learned my addr 172.21.15.28:0/2568354 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39627fc640 1 -- 172.21.15.28:0/2568354 >> v1:172.21.15.122:6789/0 conn(0x7f3964071d90 legacy=0x7f3964072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39627fc640 1 -- 172.21.15.28:0/2568354 >> v1:172.21.15.28:6789/0 conn(0x7f396413f5f0 legacy=0x7f39640ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39627fc640 1 -- 172.21.15.28:0/2568354 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396413ff60 con 0x7f396415adb0 2023-12-16T21:58:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39627fc640 1 --2- 172.21.15.28:0/2568354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396413d560 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f395800bb10 tx=0x7f395801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8a21278ee890af server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f3961ffb640 1 -- 172.21.15.28:0/2568354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39580a7ad0 con 0x7f396415adb0 2023-12-16T21:58:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f3961ffb640 1 -- 172.21.15.28:0/2568354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39580a7c30 con 0x7f396415adb0 2023-12-16T21:58:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f3961ffb640 1 -- 172.21.15.28:0/2568354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39580af440 con 0x7f396415adb0 2023-12-16T21:58:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39693e9640 1 -- 172.21.15.28:0/2568354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 msgr2=0x7f396413d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39693e9640 1 --2- 172.21.15.28:0/2568354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396413d560 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f395800bb10 tx=0x7f395801ec10 comp rx=0 tx=0).stop 2023-12-16T21:58:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39693e9640 1 -- 172.21.15.28:0/2568354 shutdown_connections 2023-12-16T21:58:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39693e9640 1 --2- 172.21.15.28:0/2568354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396413d560 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39693e9640 1 -- 172.21.15.28:0/2568354 >> 172.21.15.28:0/2568354 conn(0x7f39640b0cc0 msgr2=0x7f39640b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39693e9640 1 -- 172.21.15.28:0/2568354 shutdown_connections 2023-12-16T21:58:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.663+0000 7f39693e9640 1 -- 172.21.15.28:0/2568354 wait complete. 2023-12-16T21:58:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 Processor -- start 2023-12-16T21:58:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 -- start start 2023-12-16T21:58:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3964071d90 0x7f3964134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396413f5f0 0x7f396414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396414dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396414eab0 con 0x7f396415adb0 2023-12-16T21:58:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39640b1d60 con 0x7f396413f5f0 2023-12-16T21:58:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396414ed30 con 0x7f3964071d90 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396414dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396414dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60326/0 (socket says 172.21.15.28:60326) 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 -- 172.21.15.28:0/372919761 learned_addr learned my addr 172.21.15.28:0/372919761 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f3962ffd640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3964071d90 0x7f3964134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39627fc640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396413f5f0 0x7f396414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 -- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3964071d90 msgr2=0x7f3964134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3964071d90 0x7f3964134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 -- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396413f5f0 msgr2=0x7f396414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396413f5f0 0x7f396414d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 -- 172.21.15.28:0/372919761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395800b770 con 0x7f396415adb0 2023-12-16T21:58:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39637fe640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396414dae0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c7a0 tx=0x7f395c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f394b7fe640 1 -- 172.21.15.28:0/372919761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c00a640 con 0x7f396415adb0 2023-12-16T21:58:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f394b7fe640 1 -- 172.21.15.28:0/372919761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c00a7a0 con 0x7f396415adb0 2023-12-16T21:58:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39640b2570 con 0x7f396415adb0 2023-12-16T21:58:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f394b7fe640 1 -- 172.21.15.28:0/372919761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c00ab10 con 0x7f396415adb0 2023-12-16T21:58:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39640b2290 con 0x7f396415adb0 2023-12-16T21:58:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396414eea0 con 0x7f396415adb0 2023-12-16T21:58:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f394b7fe640 1 -- 172.21.15.28:0/372919761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f395c00ac70 con 0x7f396415adb0 2023-12-16T21:58:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.667+0000 7f394b7fe640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3950066470 0x7f3950068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.671+0000 7f394b7fe640 1 -- 172.21.15.28:0/372919761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f395c083500 con 0x7f396415adb0 2023-12-16T21:58:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.671+0000 7f3962ffd640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3950066470 0x7f3950068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.671+0000 7f3962ffd640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3950066470 0x7f3950068930 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3954000f70 tx=0x7f3954009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.675+0000 7f394b7fe640 1 -- 172.21.15.28:0/372919761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f395c050ac0 con 0x7f396415adb0 2023-12-16T21:58:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.823+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f39640693d0 con 0x7f396415adb0 2023-12-16T21:58:10.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.823+0000 7f394b7fe640 1 -- 172.21.15.28:0/372919761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f395c0540e0 con 0x7f396415adb0 2023-12-16T21:58:10.826 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3950066470 msgr2=0x7f3950068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3950066470 0x7f3950068930 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3954000f70 tx=0x7f3954009040 comp rx=0 tx=0).stop 2023-12-16T21:58:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 msgr2=0x7f396414dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396414dae0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c7a0 tx=0x7f395c00cc60 comp rx=0 tx=0).stop 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 shutdown_connections 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3950066470 0x7f3950068930 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396415adb0 0x7f396414dae0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f396413f5f0 0x7f396414d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 --2- 172.21.15.28:0/372919761 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3964071d90 0x7f3964134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 >> 172.21.15.28:0/372919761 conn(0x7f39640b0cc0 msgr2=0x7f39641597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 shutdown_connections 2023-12-16T21:58:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.827+0000 7f39693e9640 1 -- 172.21.15.28:0/372919761 wait complete. 2023-12-16T21:58:10.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:10.960 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09f2bf640 1 Processor -- start 2023-12-16T21:58:10.960 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09f2bf640 1 -- start start 2023-12-16T21:58:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09f2bf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe0980d02f0 0x7fe098167ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09f2bf640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0980aff30 con 0x7fe09815d2b0 2023-12-16T21:58:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09f2bf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0980d1ec0 con 0x7fe09815d690 2023-12-16T21:58:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09f2bf640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0980b01d0 con 0x7fe0980d02f0 2023-12-16T21:58:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09d85c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe0980d02f0 0x7fe098167ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09d05b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe09815d2b0 0x7fe0980d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43866/0 (socket says 172.21.15.28:43866) 2023-12-16T21:58:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09d05b640 1 -- 172.21.15.28:0/553221449 learned_addr learned my addr 172.21.15.28:0/553221449 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:10.962 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09d85c640 1 -- 172.21.15.28:0/553221449 >> v1:172.21.15.122:6789/0 conn(0x7fe09815d690 legacy=0x7fe0980cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:10.962 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09d85c640 1 -- 172.21.15.28:0/553221449 >> v1:172.21.15.28:6789/0 conn(0x7fe09815d2b0 legacy=0x7fe0980d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.962 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe097fff640 1 -- 172.21.15.28:0/553221449 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331007613 0 0) 0x7fe0980aff30 con 0x7fe09815d2b0 2023-12-16T21:58:10.962 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.959+0000 7fe09d85c640 1 -- 172.21.15.28:0/553221449 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0980b04b0 con 0x7fe0980d02f0 2023-12-16T21:58:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09d85c640 1 --2- 172.21.15.28:0/553221449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0980d02f0 0x7fe098167ff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe0880097c0 tx=0x7fe08801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83ee8e1f35ec81a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe097fff640 1 -- 172.21.15.28:0/553221449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0880ac410 con 0x7fe0980d02f0 2023-12-16T21:58:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe097fff640 1 -- 172.21.15.28:0/553221449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0880ac570 con 0x7fe0980d02f0 2023-12-16T21:58:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe097fff640 1 -- 172.21.15.28:0/553221449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0880ac900 con 0x7fe0980d02f0 2023-12-16T21:58:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- 172.21.15.28:0/553221449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0980d02f0 msgr2=0x7fe098167ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 --2- 172.21.15.28:0/553221449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0980d02f0 0x7fe098167ff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe0880097c0 tx=0x7fe08801c410 comp rx=0 tx=0).stop 2023-12-16T21:58:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- 172.21.15.28:0/553221449 shutdown_connections 2023-12-16T21:58:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 --2- 172.21.15.28:0/553221449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0980d02f0 0x7fe098167ff0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- 172.21.15.28:0/553221449 >> 172.21.15.28:0/553221449 conn(0x7fe09809c510 msgr2=0x7fe0980d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:10.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- 172.21.15.28:0/553221449 shutdown_connections 2023-12-16T21:58:10.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- 172.21.15.28:0/553221449 wait complete. 2023-12-16T21:58:10.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 Processor -- start 2023-12-16T21:58:10.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- start start 2023-12-16T21:58:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0980d02f0 0x7fe0980ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09815d2b0 0x7fe098155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09815d690 0x7fe0980b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0981511b0 con 0x7fe09815d690 2023-12-16T21:58:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0980cc560 con 0x7fe0980d02f0 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.963+0000 7fe09f2bf640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0980cc840 con 0x7fe09815d2b0 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09815d690 0x7fe0980b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09815d690 0x7fe0980b3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60334/0 (socket says 172.21.15.28:60334) 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 -- 172.21.15.28:0/2747449235 learned_addr learned my addr 172.21.15.28:0/2747449235 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d05b640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0980d02f0 0x7fe0980ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 -- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09815d2b0 msgr2=0x7fe098155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09c85a640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09815d2b0 0x7fe098155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09815d2b0 0x7fe098155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 -- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0980d02f0 msgr2=0x7fe0980ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0980d02f0 0x7fe0980ca9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 -- 172.21.15.28:0/2747449235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe088009420 con 0x7fe09815d690 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d05b640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0980d02f0 0x7fe0980ca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d85c640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09815d690 0x7fe0980b3680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe0880af040 tx=0x7fe0880191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe095ffb640 1 -- 172.21.15.28:0/2747449235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0880ac410 con 0x7fe09815d690 2023-12-16T21:58:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe095ffb640 1 -- 172.21.15.28:0/2747449235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe088006e20 con 0x7fe09815d690 2023-12-16T21:58:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0980ccb20 con 0x7fe09815d690 2023-12-16T21:58:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe095ffb640 1 -- 172.21.15.28:0/2747449235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0880b5840 con 0x7fe09815d690 2023-12-16T21:58:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0980cc250 con 0x7fe09815d690 2023-12-16T21:58:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0980679f0 con 0x7fe09815d690 2023-12-16T21:58:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe095ffb640 1 -- 172.21.15.28:0/2747449235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe0880b3040 con 0x7fe09815d690 2023-12-16T21:58:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe095ffb640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe084066470 0x7fe084068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe095ffb640 1 -- 172.21.15.28:0/2747449235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe08811e0c0 con 0x7fe09815d690 2023-12-16T21:58:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d05b640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe084066470 0x7fe084068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.967+0000 7fe09d05b640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe084066470 0x7fe084068930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe090000f70 tx=0x7fe090009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:10.971+0000 7fe095ffb640 1 -- 172.21.15.28:0/2747449235 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe0880eb5d0 con 0x7fe09815d690 2023-12-16T21:58:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.127+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe09815b0d0 con 0x7fe09815d690 2023-12-16T21:58:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.127+0000 7fe095ffb640 1 -- 172.21.15.28:0/2747449235 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe09815b0d0 con 0x7fe09815d690 2023-12-16T21:58:11.129 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe084066470 msgr2=0x7fe084068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe084066470 0x7fe084068930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe090000f70 tx=0x7fe090009040 comp rx=0 tx=0).stop 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09815d690 msgr2=0x7fe0980b3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09815d690 0x7fe0980b3680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe0880af040 tx=0x7fe0880191b0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 shutdown_connections 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe084066470 0x7fe084068930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09815d690 0x7fe0980b3680 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe09815d2b0 0x7fe098155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 --2- 172.21.15.28:0/2747449235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0980d02f0 0x7fe0980ca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 >> 172.21.15.28:0/2747449235 conn(0x7fe09809c510 msgr2=0x7fe0980b54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 shutdown_connections 2023-12-16T21:58:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.131+0000 7fe09f2bf640 1 -- 172.21.15.28:0/2747449235 wait complete. 2023-12-16T21:58:11.182 INFO:tasks.cephfs_test_runner:Starting test: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-12-16T21:58:11.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdf7fd640 1 Processor -- start 2023-12-16T21:58:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdf7fd640 1 -- start start 2023-12-16T21:58:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdf7fd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdfd813cf30 0x7fdfd806c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdf7fd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd815ad40 0x7fdfd80cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdf7fd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd815b3a0 con 0x7fdfd806c800 2023-12-16T21:58:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdf7fd640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd813d310 con 0x7fdfd813cf30 2023-12-16T21:58:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdf7fd640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd815b1b0 con 0x7fdfd815ad40 2023-12-16T21:58:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdd599640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdfd813cf30 0x7fdfd806c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdd599640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdfd813cf30 0x7fdfd806c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60340/0 (socket says 172.21.15.28:60340) 2023-12-16T21:58:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdd599640 1 -- 172.21.15.28:0/810885991 learned_addr learned my addr 172.21.15.28:0/810885991 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfddd9a640 1 --2- 172.21.15.28:0/810885991 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd815ad40 0x7fdfd80cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdd599640 1 -- 172.21.15.28:0/810885991 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd815ad40 msgr2=0x7fdfd80cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdd599640 1 --2- 172.21.15.28:0/810885991 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd815ad40 0x7fdfd80cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdd599640 1 -- 172.21.15.28:0/810885991 >> v1:172.21.15.122:6789/0 conn(0x7fdfd806c800 legacy=0x7fdfd815a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.306+0000 7fdfdd599640 1 -- 172.21.15.28:0/810885991 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd813f720 con 0x7fdfd813cf30 2023-12-16T21:58:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdd599640 1 --2- 172.21.15.28:0/810885991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd806c2c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009870 tx=0x7fdfcc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526fa1bdf40a5614 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfcbfff640 1 -- 172.21.15.28:0/810885991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0ab600 con 0x7fdfd813cf30 2023-12-16T21:58:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfcbfff640 1 -- 172.21.15.28:0/810885991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfcc0a7050 con 0x7fdfd813cf30 2023-12-16T21:58:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfcbfff640 1 -- 172.21.15.28:0/810885991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc0ab980 con 0x7fdfd813cf30 2023-12-16T21:58:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/810885991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 msgr2=0x7fdfd806c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/810885991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd806c2c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009870 tx=0x7fdfcc01c470 comp rx=0 tx=0).stop 2023-12-16T21:58:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/810885991 shutdown_connections 2023-12-16T21:58:11.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/810885991 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd815ad40 0x7fdfd80cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/810885991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd806c2c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/810885991 >> 172.21.15.28:0/810885991 conn(0x7fdfd806a530 msgr2=0x7fdfd806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/810885991 shutdown_connections 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/810885991 wait complete. 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 Processor -- start 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- start start 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd806c800 0x7fdfd8153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd8151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfd815ad40 0x7fdfd81517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd8153ce0 con 0x7fdfd813cf30 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd8153fc0 con 0x7fdfd815ad40 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.310+0000 7fdfdf7fd640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd81542a0 con 0x7fdfd806c800 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd8151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd8151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60352/0 (socket says 172.21.15.28:60352) 2023-12-16T21:58:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 -- 172.21.15.28:0/3634714680 learned_addr learned my addr 172.21.15.28:0/3634714680 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdd599640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd806c800 0x7fdfd8153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 -- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd806c800 msgr2=0x7fdfd8153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfddd9a640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfd815ad40 0x7fdfd81517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd806c800 0x7fdfd8153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 -- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfd815ad40 msgr2=0x7fdfd81517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfd815ad40 0x7fdfd81517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 -- 172.21.15.28:0/3634714680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc0094d0 con 0x7fdfd813cf30 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdcd98640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd8151280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400b410 tx=0x7fdfd400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfc9ffb640 1 -- 172.21.15.28:0/3634714680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4004020 con 0x7fdfd813cf30 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfc9ffb640 1 -- 172.21.15.28:0/3634714680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd4004180 con 0x7fdfd813cf30 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd8154580 con 0x7fdfd813cf30 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfc9ffb640 1 -- 172.21.15.28:0/3634714680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4011630 con 0x7fdfd813cf30 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd81538d0 con 0x7fdfd813cf30 2023-12-16T21:58:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd813a3a0 con 0x7fdfd813cf30 2023-12-16T21:58:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfc9ffb640 1 -- 172.21.15.28:0/3634714680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fdfd4011790 con 0x7fdfd813cf30 2023-12-16T21:58:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfc9ffb640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdfbc066470 0x7fdfbc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfc9ffb640 1 -- 172.21.15.28:0/3634714680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfd4083ba0 con 0x7fdfd813cf30 2023-12-16T21:58:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdd599640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdfbc066470 0x7fdfbc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.314+0000 7fdfdd599640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdfbc066470 0x7fdfbc068930 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0ad000 tx=0x7fdfcc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.318+0000 7fdfc9ffb640 1 -- 172.21.15.28:0/3634714680 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fdfd813a3a0 con 0x7fdfd813cf30 2023-12-16T21:58:11.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.474+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfd813a3a0 con 0x7fdfd813cf30 2023-12-16T21:58:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfc9ffb640 1 -- 172.21.15.28:0/3634714680 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fdfd813a3a0 con 0x7fdfd813cf30 2023-12-16T21:58:11.481 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:11.481 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdfbc066470 msgr2=0x7fdfbc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdfbc066470 0x7fdfbc068930 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc0ad000 tx=0x7fdfcc008040 comp rx=0 tx=0).stop 2023-12-16T21:58:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 msgr2=0x7fdfd8151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd8151280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdfd400b410 tx=0x7fdfd400b8d0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 shutdown_connections 2023-12-16T21:58:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdfbc066470 0x7fdfbc068930 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfd815ad40 0x7fdfd81517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd813cf30 0x7fdfd8151280 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 --2- 172.21.15.28:0/3634714680 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdfd806c800 0x7fdfd8153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 >> 172.21.15.28:0/3634714680 conn(0x7fdfd806a530 msgr2=0x7fdfd8071500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 shutdown_connections 2023-12-16T21:58:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.478+0000 7fdfdf7fd640 1 -- 172.21.15.28:0/3634714680 wait complete. 2023-12-16T21:58:11.500 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:11.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:11.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.626+0000 7f1b84a79640 1 Processor -- start 2023-12-16T21:58:11.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b84a79640 1 -- start start 2023-12-16T21:58:11.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b84a79640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b80143fa0 0x7f1b8015ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b84a79640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b80144380 con 0x7f1b800b25f0 2023-12-16T21:58:11.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b84a79640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8006b3a0 con 0x7f1b8006af40 2023-12-16T21:58:11.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b84a79640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800ceb70 con 0x7f1b80143fa0 2023-12-16T21:58:11.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7ed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1b800b25f0 0x7f1b800cefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43868/0 (socket says 172.21.15.28:43868) 2023-12-16T21:58:11.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/2008880606 learned_addr learned my addr 172.21.15.28:0/2008880606 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7e59c640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b80143fa0 0x7f1b8015ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888737251 0 0) 0x7f1b80144380 con 0x7f1b800b25f0 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b68003530 con 0x7f1b800b25f0 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 818986049 0 0) 0x7f1b68003530 con 0x7f1b800b25f0 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b80143fa0 msgr2=0x7f1b8015ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b80143fa0 0x7f1b8015ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 >> v1:172.21.15.122:6789/0 conn(0x7f1b8006af40 legacy=0x7f1b800b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8015f380 con 0x7f1b800b25f0 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f1b78001a00 con 0x7f1b800b25f0 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 >> v1:172.21.15.28:6789/0 conn(0x7f1b800b25f0 legacy=0x7f1b800cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b680039b0 0x7f1b68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b80143fa0 0x7f1b680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b68004d10 0x7f1b68005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800ceb70 con 0x7f1b68004d10 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6800ba40 con 0x7f1b680039b0 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6800bc00 con 0x7f1b80143fa0 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7ed9d640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b68004d10 0x7f1b68005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7e59c640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b680039b0 0x7f1b68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7dd9b640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b80143fa0 0x7f1b680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b80143fa0 msgr2=0x7f1b680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7ed9d640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b80143fa0 0x7f1b680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b680039b0 msgr2=0x7f1b68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7ed9d640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b680039b0 0x7f1b68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.630+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/2008880606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68003530 con 0x7f1b68004d10 2023-12-16T21:58:11.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b7ed9d640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b68004d10 0x7f1b68005190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800a800 tx=0x7f1b7801d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3228f0e5748fe2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b780072d0 con 0x7f1b68004d10 2023-12-16T21:58:11.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b780a8070 con 0x7f1b68004d10 2023-12-16T21:58:11.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b7d59a640 1 -- 172.21.15.28:0/2008880606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78007690 con 0x7f1b68004d10 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 -- 172.21.15.28:0/2008880606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b68004d10 msgr2=0x7f1b68005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 --2- 172.21.15.28:0/2008880606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b68004d10 0x7f1b68005190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800a800 tx=0x7f1b7801d300 comp rx=0 tx=0).stop 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/2008880606 reap_dead start 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 -- 172.21.15.28:0/2008880606 shutdown_connections 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 -- 172.21.15.28:0/2008880606 >> 172.21.15.28:0/2008880606 conn(0x7f1b800b54d0 msgr2=0x7f1b800b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 -- 172.21.15.28:0/2008880606 shutdown_connections 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 -- 172.21.15.28:0/2008880606 wait complete. 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 Processor -- start 2023-12-16T21:58:11.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 -- start start 2023-12-16T21:58:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b68004d10 0x7f1b800a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.634+0000 7f1b84a79640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800a88d0 0x7f1b8013eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b84a79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800cc590 0x7f1b800cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b84a79640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800cb5f0 con 0x7f1b800cc590 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b84a79640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800cc100 con 0x7f1b800a88d0 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b84a79640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800a9660 con 0x7f1b68004d10 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800cc590 0x7f1b800cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800cc590 0x7f1b800cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60360/0 (socket says 172.21.15.28:60360) 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/4277637916 learned_addr learned my addr 172.21.15.28:0/4277637916 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7dd9b640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800a88d0 0x7f1b8013eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7e59c640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b68004d10 0x7f1b800a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b68004d10 msgr2=0x7f1b800a8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b68004d10 0x7f1b800a8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800a88d0 msgr2=0x7f1b8013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800a88d0 0x7f1b8013eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 -- 172.21.15.28:0/4277637916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b800cb7d0 con 0x7f1b800cc590 2023-12-16T21:58:11.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7dd9b640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800a88d0 0x7f1b8013eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7ed9d640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800cc590 0x7f1b800cca10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1b7801d130 tx=0x7f1b68005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b6effd640 1 -- 172.21.15.28:0/4277637916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b780011d0 con 0x7f1b800cc590 2023-12-16T21:58:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b6effd640 1 -- 172.21.15.28:0/4277637916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b780017c0 con 0x7f1b800cc590 2023-12-16T21:58:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b800cbab0 con 0x7f1b800cc590 2023-12-16T21:58:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b6effd640 1 -- 172.21.15.28:0/4277637916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78004b70 con 0x7f1b800cc590 2023-12-16T21:58:11.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b800a9350 con 0x7f1b800cc590 2023-12-16T21:58:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b800cc270 con 0x7f1b800cc590 2023-12-16T21:58:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b6effd640 1 -- 172.21.15.28:0/4277637916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1b780aa030 con 0x7f1b800cc590 2023-12-16T21:58:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b6effd640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1b600664a0 0x7f1b60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b6effd640 1 -- 172.21.15.28:0/4277637916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b7811d770 con 0x7f1b800cc590 2023-12-16T21:58:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7e59c640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1b600664a0 0x7f1b60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.638+0000 7f1b7e59c640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1b600664a0 0x7f1b60068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009a30 tx=0x7f1b70008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.642+0000 7f1b6effd640 1 -- 172.21.15.28:0/4277637916 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f1b780eabf0 con 0x7f1b800cc590 2023-12-16T21:58:11.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.798+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b800674d0 con 0x7f1b800cc590 2023-12-16T21:58:11.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.798+0000 7f1b6effd640 1 -- 172.21.15.28:0/4277637916 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b780ee210 con 0x7f1b800cc590 2023-12-16T21:58:11.802 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1b600664a0 msgr2=0x7f1b60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1b600664a0 0x7f1b60068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009a30 tx=0x7f1b70008040 comp rx=0 tx=0).stop 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800cc590 msgr2=0x7f1b800cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800cc590 0x7f1b800cca10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1b7801d130 tx=0x7f1b68005800 comp rx=0 tx=0).stop 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 shutdown_connections 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1b600664a0 0x7f1b60068960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800cc590 0x7f1b800cca10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b800a88d0 0x7f1b8013eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 --2- 172.21.15.28:0/4277637916 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1b68004d10 0x7f1b800a8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 >> 172.21.15.28:0/4277637916 conn(0x7f1b800b54d0 msgr2=0x7f1b80144380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 shutdown_connections 2023-12-16T21:58:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.802+0000 7f1b84a79640 1 -- 172.21.15.28:0/4277637916 wait complete. 2023-12-16T21:58:11.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9d06156640 1 Processor -- start 2023-12-16T21:58:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9d06156640 1 -- start start 2023-12-16T21:58:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9d06156640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9d00158ca0 0x7f9d0015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9d06156640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9d000b42d0 0x7f9d000b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9d06156640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d000b4c60 con 0x7f9d000cba10 2023-12-16T21:58:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9d06156640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0015b670 con 0x7f9d000b42d0 2023-12-16T21:58:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9d06156640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d0015c510 con 0x7f9d00158ca0 2023-12-16T21:58:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9cfffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9d000b42d0 0x7f9d000b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9cfffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9d000b42d0 0x7f9d000b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60376/0 (socket says 172.21.15.28:60376) 2023-12-16T21:58:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9cfffff640 1 -- 172.21.15.28:0/3416260327 learned_addr learned my addr 172.21.15.28:0/3416260327 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9cff7fe640 1 --2- 172.21.15.28:0/3416260327 >> v2:172.21.15.122:3300/0 conn(0x7f9d00158ca0 0x7f9d0015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9cfffff640 1 -- 172.21.15.28:0/3416260327 >> v2:172.21.15.122:3300/0 conn(0x7f9d00158ca0 msgr2=0x7f9d0015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9cfffff640 1 --2- 172.21.15.28:0/3416260327 >> v2:172.21.15.122:3300/0 conn(0x7f9d00158ca0 0x7f9d0015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9cfffff640 1 -- 172.21.15.28:0/3416260327 >> v1:172.21.15.122:6789/0 conn(0x7f9d000cba10 legacy=0x7f9d0015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.938+0000 7f9cfffff640 1 -- 172.21.15.28:0/3416260327 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0015f410 con 0x7f9d000b42d0 2023-12-16T21:58:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cfffff640 1 --2- 172.21.15.28:0/3416260327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000b42d0 0x7f9d000b4720 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9d00067f70 tx=0x7f9cf801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2281402e1d34bdfd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cfe7fc640 1 -- 172.21.15.28:0/3416260327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80a5820 con 0x7f9d000b42d0 2023-12-16T21:58:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cfe7fc640 1 -- 172.21.15.28:0/3416260327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf80a5980 con 0x7f9d000b42d0 2023-12-16T21:58:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cfe7fc640 1 -- 172.21.15.28:0/3416260327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80a5d10 con 0x7f9d000b42d0 2023-12-16T21:58:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- 172.21.15.28:0/3416260327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000b42d0 msgr2=0x7f9d000b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 --2- 172.21.15.28:0/3416260327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000b42d0 0x7f9d000b4720 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9d00067f70 tx=0x7f9cf801c960 comp rx=0 tx=0).stop 2023-12-16T21:58:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- 172.21.15.28:0/3416260327 shutdown_connections 2023-12-16T21:58:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 --2- 172.21.15.28:0/3416260327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000b42d0 0x7f9d000b4720 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 --2- 172.21.15.28:0/3416260327 >> v2:172.21.15.122:3300/0 conn(0x7f9d00158ca0 0x7f9d0015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- 172.21.15.28:0/3416260327 >> 172.21.15.28:0/3416260327 conn(0x7f9d000b0cc0 msgr2=0x7f9d000b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- 172.21.15.28:0/3416260327 shutdown_connections 2023-12-16T21:58:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- 172.21.15.28:0/3416260327 wait complete. 2023-12-16T21:58:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 Processor -- start 2023-12-16T21:58:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- start start 2023-12-16T21:58:11.947 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d000b42d0 0x7f9d0014c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.947 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000cba10 0x7f9d000b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d00158ca0 0x7f9d00165610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00165180 con 0x7f9d000cba10 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00164570 con 0x7f9d00158ca0 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9d06156640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00164850 con 0x7f9d000b42d0 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cfeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000cba10 0x7f9d000b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cfeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000cba10 0x7f9d000b3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60386/0 (socket says 172.21.15.28:60386) 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cfeffd640 1 -- 172.21.15.28:0/3235119833 learned_addr learned my addr 172.21.15.28:0/3235119833 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cff7fe640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d000b42d0 0x7f9d0014c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.942+0000 7f9cfffff640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d00158ca0 0x7f9d00165610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9cfeffd640 1 -- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d000b42d0 msgr2=0x7f9d0014c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9cfeffd640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d000b42d0 0x7f9d0014c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9cfeffd640 1 -- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d00158ca0 msgr2=0x7f9d00165610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:11.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9cfeffd640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d00158ca0 0x7f9d00165610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:11.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9cfeffd640 1 -- 172.21.15.28:0/3235119833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf8009b30 con 0x7f9d000cba10 2023-12-16T21:58:11.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9cfeffd640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000cba10 0x7f9d000b3a10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400b3e0 tx=0x7f9cf400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3235119833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4011830 con 0x7f9d000cba10 2023-12-16T21:58:11.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3235119833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4011990 con 0x7f9d000cba10 2023-12-16T21:58:11.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d00164b30 con 0x7f9d000cba10 2023-12-16T21:58:11.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3235119833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4011d80 con 0x7f9d000cba10 2023-12-16T21:58:11.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0014c890 con 0x7f9d000cba10 2023-12-16T21:58:11.950 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0015f010 con 0x7f9d000cba10 2023-12-16T21:58:11.950 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3235119833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f9cf401a400 con 0x7f9d000cba10 2023-12-16T21:58:11.950 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9ce7fff640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9cec066470 0x7f9cec068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:11.950 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3235119833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cf4083b50 con 0x7f9d000cba10 2023-12-16T21:58:11.950 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9cff7fe640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9cec066470 0x7f9cec068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:11.950 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.946+0000 7f9cff7fe640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9cec066470 0x7f9cec068930 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0007a60 tx=0x7f9cf0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:11.954 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:11.950+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3235119833 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f9cf4051110 con 0x7f9d000cba10 2023-12-16T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.106+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d000693d0 con 0x7f9d000cba10 2023-12-16T21:58:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.106+0000 7f9ce7fff640 1 -- 172.21.15.28:0/3235119833 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9cf4054730 con 0x7f9d000cba10 2023-12-16T21:58:12.110 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9cec066470 msgr2=0x7f9cec068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9cec066470 0x7f9cec068930 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0007a60 tx=0x7f9cf0008040 comp rx=0 tx=0).stop 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000cba10 msgr2=0x7f9d000b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000cba10 0x7f9d000b3a10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400b3e0 tx=0x7f9cf400b8a0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 shutdown_connections 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9cec066470 0x7f9cec068930 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d00158ca0 0x7f9d00165610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d000cba10 0x7f9d000b3a10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 --2- 172.21.15.28:0/3235119833 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9d000b42d0 0x7f9d0014c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 >> 172.21.15.28:0/3235119833 conn(0x7f9d000b0cc0 msgr2=0x7f9d00151450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 shutdown_connections 2023-12-16T21:58:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.110+0000 7f9d06156640 1 -- 172.21.15.28:0/3235119833 wait complete. 2023-12-16T21:58:12.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.250+0000 7f3cc935f640 1 Processor -- start 2023-12-16T21:58:12.255 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc935f640 1 -- start start 2023-12-16T21:58:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc935f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3cc40d02f0 0x7f3cc4167ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc935f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc40aff30 con 0x7f3cc415d2b0 2023-12-16T21:58:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc935f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cc40d1ec0 con 0x7f3cc415d690 2023-12-16T21:58:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc935f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cc40b01d0 con 0x7f3cc40d02f0 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc2ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3cc415d2b0 0x7f3cc40d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43878/0 (socket says 172.21.15.28:43878) 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc2ffd640 1 -- 172.21.15.28:0/3982138875 learned_addr learned my addr 172.21.15.28:0/3982138875 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc37fe640 1 --2- 172.21.15.28:0/3982138875 >> v2:172.21.15.122:3300/0 conn(0x7f3cc40d02f0 0x7f3cc4167ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732008281 0 0) 0x7f3cc40aff30 con 0x7f3cc415d2b0 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cac003530 con 0x7f3cc415d2b0 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2764817237 0 0) 0x7f3cac003530 con 0x7f3cc415d2b0 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 >> v2:172.21.15.122:3300/0 conn(0x7f3cc40d02f0 msgr2=0x7f3cc4167ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 --2- 172.21.15.28:0/3982138875 >> v2:172.21.15.122:3300/0 conn(0x7f3cc40d02f0 0x7f3cc4167ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 >> v1:172.21.15.122:6789/0 conn(0x7f3cc415d690 legacy=0x7f3cc40cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc40b04b0 con 0x7f3cc415d2b0 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f3cb4001680 con 0x7f3cc415d2b0 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 >> v1:172.21.15.28:6789/0 conn(0x7f3cc415d2b0 legacy=0x7f3cc40d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cac003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cac0047d0 0x7f3cac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cac005190 0x7f3cac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc40b01d0 con 0x7f3cac0039d0 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac00c090 con 0x7f3cac005190 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac00c250 con 0x7f3cac0047d0 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc2ffd640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cac003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc37fe640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cac005190 0x7f3cac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc27fc640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cac0047d0 0x7f3cac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc2ffd640 1 -- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cac0047d0 msgr2=0x7f3cac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc2ffd640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cac0047d0 0x7f3cac004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc2ffd640 1 -- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cac005190 msgr2=0x7f3cac00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc2ffd640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cac005190 0x7f3cac00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc2ffd640 1 -- 172.21.15.28:0/3982138875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac003530 con 0x7f3cac0039d0 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc37fe640 1 -- 172.21.15.28:0/3982138875 reap_dead start 2023-12-16T21:58:12.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.254+0000 7f3cc27fc640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cac0047d0 0x7f3cac004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc2ffd640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cac003df0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400c110 tx=0x7f3cb401caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845a40d27144c7a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4005ae0 con 0x7f3cac0039d0 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb4005c40 con 0x7f3cac0039d0 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc1ffb640 1 -- 172.21.15.28:0/3982138875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4007070 con 0x7f3cac0039d0 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- 172.21.15.28:0/3982138875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 msgr2=0x7f3cac003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cac003df0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400c110 tx=0x7f3cb401caf0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- 172.21.15.28:0/3982138875 shutdown_connections 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 --2- 172.21.15.28:0/3982138875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cac003df0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- 172.21.15.28:0/3982138875 >> 172.21.15.28:0/3982138875 conn(0x7f3cc409c510 msgr2=0x7f3cc40d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- 172.21.15.28:0/3982138875 shutdown_connections 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- 172.21.15.28:0/3982138875 wait complete. 2023-12-16T21:58:12.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 Processor -- start 2023-12-16T21:58:12.263 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- start start 2023-12-16T21:58:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cc4148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cc40b6280 0x7f3cc40c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.264 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40c8c70 0x7f3cc40c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc40ca6e0 con 0x7f3cac0039d0 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc40b5df0 con 0x7f3cc40c8c70 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc935f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc40c9660 con 0x7f3cc40b6280 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cc4148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cc4148a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60410/0 (socket says 172.21.15.28:60410) 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc2ffd640 1 -- 172.21.15.28:0/340443816 learned_addr learned my addr 172.21.15.28:0/340443816 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc27fc640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cc40b6280 0x7f3cc40c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.258+0000 7f3cc37fe640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40c8c70 0x7f3cc40c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc2ffd640 1 -- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cc40b6280 msgr2=0x7f3cc40c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc2ffd640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cc40b6280 0x7f3cc40c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc2ffd640 1 -- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40c8c70 msgr2=0x7f3cc40c9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc2ffd640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40c8c70 0x7f3cc40c9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc2ffd640 1 -- 172.21.15.28:0/340443816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc4148fd0 con 0x7f3cac0039d0 2023-12-16T21:58:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc2ffd640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cc4148a90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3cb401c560 tx=0x7f3cac005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cab7fe640 1 -- 172.21.15.28:0/340443816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb40b4070 con 0x7f3cac0039d0 2023-12-16T21:58:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cab7fe640 1 -- 172.21.15.28:0/340443816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cb4001480 con 0x7f3cac0039d0 2023-12-16T21:58:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc40b8ee0 con 0x7f3cac0039d0 2023-12-16T21:58:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cab7fe640 1 -- 172.21.15.28:0/340443816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb40072f0 con 0x7f3cac0039d0 2023-12-16T21:58:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc40cac10 con 0x7f3cac0039d0 2023-12-16T21:58:12.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc40679f0 con 0x7f3cac0039d0 2023-12-16T21:58:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cab7fe640 1 -- 172.21.15.28:0/340443816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f3cb4007450 con 0x7f3cac0039d0 2023-12-16T21:58:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cab7fe640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3cb00664a0 0x7f3cb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cab7fe640 1 -- 172.21.15.28:0/340443816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3cb411d3b0 con 0x7f3cac0039d0 2023-12-16T21:58:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc27fc640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3cb00664a0 0x7f3cb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.262+0000 7f3cc27fc640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3cb00664a0 0x7f3cb0068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40cd0d0 tx=0x7f3cb8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:12.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.266+0000 7f3cab7fe640 1 -- 172.21.15.28:0/340443816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3cb40ea7b0 con 0x7f3cac0039d0 2023-12-16T21:58:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.414+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3cc40693d0 con 0x7f3cac0039d0 2023-12-16T21:58:12.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.414+0000 7f3cab7fe640 1 -- 172.21.15.28:0/340443816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3cb40eddd0 con 0x7f3cac0039d0 2023-12-16T21:58:12.418 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:12.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3cb00664a0 msgr2=0x7f3cb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3cb00664a0 0x7f3cb0068960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3cc40cd0d0 tx=0x7f3cb8009040 comp rx=0 tx=0).stop 2023-12-16T21:58:12.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 msgr2=0x7f3cc4148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cc4148a90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3cb401c560 tx=0x7f3cac005800 comp rx=0 tx=0).stop 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 shutdown_connections 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3cb00664a0 0x7f3cb0068960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40c8c70 0x7f3cc40c9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3cc40b6280 0x7f3cc40c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 --2- 172.21.15.28:0/340443816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac0039d0 0x7f3cc4148a90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 >> 172.21.15.28:0/340443816 conn(0x7f3cc409c510 msgr2=0x7f3cc40d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 shutdown_connections 2023-12-16T21:58:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.418+0000 7f3cc935f640 1 -- 172.21.15.28:0/340443816 wait complete. 2023-12-16T21:58:12.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic' 2023-12-16T21:58:12.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc846256640 1 Processor -- start 2023-12-16T21:58:12.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc846256640 1 -- start start 2023-12-16T21:58:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc846256640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8400d01d0 0x7fc8400d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc846256640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8400d0d30 con 0x7fc8400d1780 2023-12-16T21:58:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc846256640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8400d12f0 con 0x7fc8400b3bc0 2023-12-16T21:58:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc846256640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc840160f80 con 0x7fc8400d01d0 2023-12-16T21:58:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83effd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc8400d1780 0x7fc8400cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43894/0 (socket says 172.21.15.28:43894) 2023-12-16T21:58:12.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83effd640 1 -- 172.21.15.28:0/2991642714 learned_addr learned my addr 172.21.15.28:0/2991642714 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83ffff640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8400d01d0 0x7fc8400d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216125351 0 0) 0x7fc8400d0d30 con 0x7fc8400d1780 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc828003530 con 0x7fc8400d1780 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 980527913 0 0) 0x7fc828003530 con 0x7fc8400d1780 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8400d01d0 msgr2=0x7fc8400d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8400d01d0 0x7fc8400d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 >> v1:172.21.15.122:6789/0 conn(0x7fc8400b3bc0 legacy=0x7fc8400b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400cecc0 con 0x7fc8400d1780 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fc834001680 con 0x7fc8400d1780 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 >> v1:172.21.15.28:6789/0 conn(0x7fc8400d1780 legacy=0x7fc8400cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8280039b0 0x7fc828003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8400d01d0 0x7fc8280047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc828004d10 0x7fc828005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc840160f80 con 0x7fc8280039b0 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82800ba70 con 0x7fc828004d10 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82800bc30 con 0x7fc8400d01d0 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83f7fe640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8280039b0 0x7fc828003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83effd640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8400d01d0 0x7fc8280047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83ffff640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc828004d10 0x7fc828005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83f7fe640 1 -- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8400d01d0 msgr2=0x7fc8280047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83f7fe640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8400d01d0 0x7fc8280047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83f7fe640 1 -- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc828004d10 msgr2=0x7fc828005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83f7fe640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc828004d10 0x7fc828005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.558+0000 7fc83f7fe640 1 -- 172.21.15.28:0/2991642714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828003530 con 0x7fc8280039b0 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc83f7fe640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8280039b0 0x7fc828003dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc830001750 tx=0x7fc83001c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd58ccd74c80dfeb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8300052d0 con 0x7fc8280039b0 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8300a7050 con 0x7fc8280039b0 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc83e7fc640 1 -- 172.21.15.28:0/2991642714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830003600 con 0x7fc8280039b0 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc846256640 1 -- 172.21.15.28:0/2991642714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8280039b0 msgr2=0x7fc828003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc846256640 1 --2- 172.21.15.28:0/2991642714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8280039b0 0x7fc828003dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc830001750 tx=0x7fc83001c390 comp rx=0 tx=0).stop 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc83ffff640 1 -- 172.21.15.28:0/2991642714 reap_dead start 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc846256640 1 -- 172.21.15.28:0/2991642714 shutdown_connections 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc846256640 1 -- 172.21.15.28:0/2991642714 >> 172.21.15.28:0/2991642714 conn(0x7fc84009c430 msgr2=0x7fc8400d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc846256640 1 -- 172.21.15.28:0/2991642714 shutdown_connections 2023-12-16T21:58:12.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.562+0000 7fc846256640 1 -- 172.21.15.28:0/2991642714 wait complete. 2023-12-16T21:58:12.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 Processor -- start 2023-12-16T21:58:12.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 -- start start 2023-12-16T21:58:12.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8280039b0 0x7fc8400b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400ccd60 0x7fc840160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8401514c0 0x7fc840160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8400b8780 con 0x7fc8400ccd60 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc840151350 con 0x7fc8280039b0 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc840151dc0 con 0x7fc8401514c0 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400ccd60 0x7fc840160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400ccd60 0x7fc840160130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60432/0 (socket says 172.21.15.28:60432) 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 -- 172.21.15.28:0/3205624217 learned_addr learned my addr 172.21.15.28:0/3205624217 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83f7fe640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8280039b0 0x7fc8400b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83ffff640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8401514c0 0x7fc840160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 -- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8401514c0 msgr2=0x7fc840160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8401514c0 0x7fc840160670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 -- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8280039b0 msgr2=0x7fc8400b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8280039b0 0x7fc8400b7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:12.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 -- 172.21.15.28:0/3205624217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400b84a0 con 0x7fc8400ccd60 2023-12-16T21:58:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83f7fe640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8280039b0 0x7fc8400b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc83effd640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400ccd60 0x7fc840160130 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc834007290 tx=0x7fc834007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc827fff640 1 -- 172.21.15.28:0/3205624217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400f9c0 con 0x7fc8400ccd60 2023-12-16T21:58:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc827fff640 1 -- 172.21.15.28:0/3205624217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83400fb20 con 0x7fc8400ccd60 2023-12-16T21:58:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8400ca3d0 con 0x7fc8400ccd60 2023-12-16T21:58:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc827fff640 1 -- 172.21.15.28:0/3205624217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400fed0 con 0x7fc8400ccd60 2023-12-16T21:58:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8400ca990 con 0x7fc8400ccd60 2023-12-16T21:58:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.566+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8400ca210 con 0x7fc8400ccd60 2023-12-16T21:58:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.570+0000 7fc827fff640 1 -- 172.21.15.28:0/3205624217 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fc83401d460 con 0x7fc8400ccd60 2023-12-16T21:58:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.570+0000 7fc827fff640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc82c066470 0x7fc82c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.570+0000 7fc827fff640 1 -- 172.21.15.28:0/3205624217 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8340841f0 con 0x7fc8400ccd60 2023-12-16T21:58:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.570+0000 7fc83f7fe640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc82c066470 0x7fc82c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.570+0000 7fc83f7fe640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc82c066470 0x7fc82c068930 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc8280057b0 tx=0x7fc8300011a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:12.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.574+0000 7fc827fff640 1 -- 172.21.15.28:0/3205624217 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fc8340515e0 con 0x7fc8400ccd60 2023-12-16T21:58:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:12.786+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic"]} v 0) v1 -- 0x7fc8400674d0 con 0x7fc8400ccd60 2023-12-16T21:58:13.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.694+0000 7fc827fff640 1 -- 172.21.15.28:0/3205624217 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic"]}]=0 v51) v1 ==== 147+0+0 (secure 0 0 0) 0x7fc834054c00 con 0x7fc8400ccd60 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc82c066470 msgr2=0x7fc82c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc82c066470 0x7fc82c068930 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc8280057b0 tx=0x7fc8300011a0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400ccd60 msgr2=0x7fc840160130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400ccd60 0x7fc840160130 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc834007290 tx=0x7fc834007750 comp rx=0 tx=0).stop 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 shutdown_connections 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc82c066470 0x7fc82c068930 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc8401514c0 0x7fc840160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8400ccd60 0x7fc840160130 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 --2- 172.21.15.28:0/3205624217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8280039b0 0x7fc8400b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 >> 172.21.15.28:0/3205624217 conn(0x7fc84009c430 msgr2=0x7fc8400d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 shutdown_connections 2023-12-16T21:58:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.698+0000 7fc846256640 1 -- 172.21.15.28:0/3205624217 wait complete. 2023-12-16T21:58:13.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-16T21:58:13.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:58:13.826 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:58:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fe19e3640 1 Processor -- start 2023-12-16T21:58:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fe19e3640 1 -- start start 2023-12-16T21:58:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fe19e3640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc071590 0x7f4fdc071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fe19e3640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fdc0b2fd0 0x7f4fdc0b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fe19e3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fdc06ff20 con 0x7f4fdc143b30 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fe19e3640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fdc071f50 con 0x7f4fdc0b2fd0 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fe19e3640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc072a70 con 0x7f4fdc071590 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fdb7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fdc0b2fd0 0x7f4fdc0b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fdb7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fdc0b2fd0 0x7f4fdc0b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60448/0 (socket says 172.21.15.28:60448) 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fdb7fe640 1 -- 172.21.15.28:0/1454483235 learned_addr learned my addr 172.21.15.28:0/1454483235 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fda7fc640 1 --2- 172.21.15.28:0/1454483235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc071590 0x7f4fdc071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fdb7fe640 1 -- 172.21.15.28:0/1454483235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc071590 msgr2=0x7f4fdc071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fdb7fe640 1 --2- 172.21.15.28:0/1454483235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc071590 0x7f4fdc071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fdb7fe640 1 -- 172.21.15.28:0/1454483235 >> v1:172.21.15.122:6789/0 conn(0x7f4fdc143b30 legacy=0x7f4fdc072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fdb7fe640 1 -- 172.21.15.28:0/1454483235 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc0cec90 con 0x7f4fdc0b2fd0 2023-12-16T21:58:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.854+0000 7f4fda7fc640 1 --2- 172.21.15.28:0/1454483235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc071590 0x7f4fdc071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fdb7fe640 1 --2- 172.21.15.28:0/1454483235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc0b2fd0 0x7f4fdc0b3450 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009e50 tx=0x7f4fd401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55ae615f007922b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fd9ffb640 1 -- 172.21.15.28:0/1454483235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd40a5820 con 0x7f4fdc0b2fd0 2023-12-16T21:58:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fd9ffb640 1 -- 172.21.15.28:0/1454483235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd40a5980 con 0x7f4fdc0b2fd0 2023-12-16T21:58:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fd9ffb640 1 -- 172.21.15.28:0/1454483235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd40a5d10 con 0x7f4fdc0b2fd0 2023-12-16T21:58:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1454483235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc0b2fd0 msgr2=0x7f4fdc0b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1454483235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc0b2fd0 0x7f4fdc0b3450 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009e50 tx=0x7f4fd401c9a0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1454483235 shutdown_connections 2023-12-16T21:58:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1454483235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc0b2fd0 0x7f4fdc0b3450 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1454483235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc071590 0x7f4fdc071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1454483235 >> 172.21.15.28:0/1454483235 conn(0x7f4fdc06d510 msgr2=0x7f4fdc06d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1454483235 shutdown_connections 2023-12-16T21:58:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1454483235 wait complete. 2023-12-16T21:58:13.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 Processor -- start 2023-12-16T21:58:13.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.858+0000 7f4fe19e3640 1 -- start start 2023-12-16T21:58:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc071590 0x7f4fdc15d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fdc0b2fd0 0x7f4fdc158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc143b30 0x7f4fdc15b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc15cf00 con 0x7f4fdc071590 2023-12-16T21:58:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc159e80 con 0x7f4fdc0b2fd0 2023-12-16T21:58:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc15d860 con 0x7f4fdc143b30 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc071590 0x7f4fdc15d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc071590 0x7f4fdc15d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60462/0 (socket says 172.21.15.28:60462) 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 -- 172.21.15.28:0/1928952674 learned_addr learned my addr 172.21.15.28:0/1928952674 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fda7fc640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fdc0b2fd0 0x7f4fdc158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdb7fe640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc143b30 0x7f4fdc15b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 -- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc143b30 msgr2=0x7f4fdc15b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc143b30 0x7f4fdc15b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 -- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fdc0b2fd0 msgr2=0x7f4fdc158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fdc0b2fd0 0x7f4fdc158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 -- 172.21.15.28:0/1928952674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd4009b30 con 0x7f4fdc071590 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdb7fe640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc143b30 0x7f4fdc15b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fdaffd640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc071590 0x7f4fdc15d320 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00c450 tx=0x7f4fcc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:13.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fd3fff640 1 -- 172.21.15.28:0/1928952674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc012830 con 0x7f4fdc071590 2023-12-16T21:58:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fd3fff640 1 -- 172.21.15.28:0/1928952674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fcc0129b0 con 0x7f4fdc071590 2023-12-16T21:58:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc15c2a0 con 0x7f4fdc071590 2023-12-16T21:58:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fd3fff640 1 -- 172.21.15.28:0/1928952674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc012d60 con 0x7f4fdc071590 2023-12-16T21:58:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc15c560 con 0x7f4fdc071590 2023-12-16T21:58:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc15d040 con 0x7f4fdc071590 2023-12-16T21:58:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fd3fff640 1 -- 172.21.15.28:0/1928952674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4fcc01c460 con 0x7f4fdc071590 2023-12-16T21:58:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fd3fff640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4fbc066470 0x7f4fbc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fd3fff640 1 -- 172.21.15.28:0/1928952674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fcc083c60 con 0x7f4fdc071590 2023-12-16T21:58:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.862+0000 7f4fda7fc640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4fbc066470 0x7f4fbc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.866+0000 7f4fda7fc640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4fbc066470 0x7f4fbc068930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4006fd0 tx=0x7f4fc4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:13.866+0000 7f4fd3fff640 1 -- 172.21.15.28:0/1928952674 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4fdc15d040 con 0x7f4fdc071590 2023-12-16T21:58:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.022+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4fdc15d040 con 0x7f4fdc071590 2023-12-16T21:58:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fd3fff640 1 -- 172.21.15.28:0/1928952674 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4fdc15d040 con 0x7f4fdc071590 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4fbc066470 msgr2=0x7f4fbc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4fbc066470 0x7f4fbc068930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4006fd0 tx=0x7f4fc4008040 comp rx=0 tx=0).stop 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc071590 msgr2=0x7f4fdc15d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc071590 0x7f4fdc15d320 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00c450 tx=0x7f4fcc00c910 comp rx=0 tx=0).stop 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 shutdown_connections 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4fbc066470 0x7f4fbc068930 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4fdc143b30 0x7f4fdc15b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fdc0b2fd0 0x7f4fdc158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 --2- 172.21.15.28:0/1928952674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fdc071590 0x7f4fdc15d320 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 >> 172.21.15.28:0/1928952674 conn(0x7f4fdc06d510 msgr2=0x7f4fdc072360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 shutdown_connections 2023-12-16T21:58:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.030+0000 7f4fe19e3640 1 -- 172.21.15.28:0/1928952674 wait complete. 2023-12-16T21:58:14.052 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-16T21:58:14.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:58:14.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:58:14.186 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-16T21:58:14.186 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:58:14.186 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:14.191 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:58:14.191 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:14.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b54efc640 1 Processor -- start 2023-12-16T21:58:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b54efc640 1 -- start start 2023-12-16T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b54efc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6b5006b640 0x7f6b5006c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b54efc640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b500d1d30 con 0x7f6b5015d250 2023-12-16T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b54efc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b500cf0c0 con 0x7f6b5009a0e0 2023-12-16T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b54efc640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b500cf360 con 0x7f6b5006b640 2023-12-16T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b4e59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6b5015d250 0x7f6b500999d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43900/0 (socket says 172.21.15.28:43900) 2023-12-16T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b4e59c640 1 -- 172.21.15.28:0/2671857818 learned_addr learned my addr 172.21.15.28:0/2671857818 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b4ed9d640 1 --2- 172.21.15.28:0/2671857818 >> v2:172.21.15.28:3300/0 conn(0x7f6b5006b640 0x7f6b5006c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b4d59a640 1 -- 172.21.15.28:0/2671857818 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528269641 0 0) 0x7f6b500d1d30 con 0x7f6b5015d250 2023-12-16T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b4d59a640 1 -- 172.21.15.28:0/2671857818 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b38003530 con 0x7f6b5015d250 2023-12-16T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b4ed9d640 1 -- 172.21.15.28:0/2671857818 >> v1:172.21.15.122:6789/0 conn(0x7f6b5009a0e0 legacy=0x7f6b500b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b4ed9d640 1 -- 172.21.15.28:0/2671857818 >> v1:172.21.15.28:6789/0 conn(0x7f6b5015d250 legacy=0x7f6b500999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.406+0000 7f6b4ed9d640 1 -- 172.21.15.28:0/2671857818 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b500cf640 con 0x7f6b5006b640 2023-12-16T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b4ed9d640 1 --2- 172.21.15.28:0/2671857818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5006b640 0x7f6b5006c520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009d90 tx=0x7f6b4801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9b1069883b7a43 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b4d59a640 1 -- 172.21.15.28:0/2671857818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b480a59a0 con 0x7f6b5006b640 2023-12-16T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b4d59a640 1 -- 172.21.15.28:0/2671857818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b480a5b00 con 0x7f6b5006b640 2023-12-16T21:58:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b4d59a640 1 -- 172.21.15.28:0/2671857818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b480a5e90 con 0x7f6b5006b640 2023-12-16T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- 172.21.15.28:0/2671857818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5006b640 msgr2=0x7f6b5006c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 --2- 172.21.15.28:0/2671857818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5006b640 0x7f6b5006c520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009d90 tx=0x7f6b4801c8e0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- 172.21.15.28:0/2671857818 shutdown_connections 2023-12-16T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 --2- 172.21.15.28:0/2671857818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5006b640 0x7f6b5006c520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- 172.21.15.28:0/2671857818 >> 172.21.15.28:0/2671857818 conn(0x7f6b500d0f30 msgr2=0x7f6b500d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- 172.21.15.28:0/2671857818 shutdown_connections 2023-12-16T21:58:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- 172.21.15.28:0/2671857818 wait complete. 2023-12-16T21:58:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 Processor -- start 2023-12-16T21:58:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- start start 2023-12-16T21:58:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b5006b640 0x7f6b500b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5009a0e0 0x7f6b50157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5015d250 0x7f6b500c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b500b6d50 con 0x7f6b5015d250 2023-12-16T21:58:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b500b4790 con 0x7f6b5009a0e0 2023-12-16T21:58:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b54efc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b500c5d40 con 0x7f6b5006b640 2023-12-16T21:58:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b4ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5015d250 0x7f6b500c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b4ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5015d250 0x7f6b500c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60480/0 (socket says 172.21.15.28:60480) 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.410+0000 7f6b4ed9d640 1 -- 172.21.15.28:0/4022570226 learned_addr learned my addr 172.21.15.28:0/4022570226 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4dd9b640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5009a0e0 0x7f6b50157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4e59c640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b5006b640 0x7f6b500b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4ed9d640 1 -- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b5006b640 msgr2=0x7f6b500b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4ed9d640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b5006b640 0x7f6b500b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4ed9d640 1 -- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5009a0e0 msgr2=0x7f6b50157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4ed9d640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5009a0e0 0x7f6b50157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4ed9d640 1 -- 172.21.15.28:0/4022570226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b48009a70 con 0x7f6b5015d250 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4ed9d640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5015d250 0x7f6b500c6ad0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6b4801a510 tx=0x7f6b48019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b3effd640 1 -- 172.21.15.28:0/4022570226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b480add90 con 0x7f6b5015d250 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b3effd640 1 -- 172.21.15.28:0/4022570226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b48009450 con 0x7f6b5015d250 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b500b49e0 con 0x7f6b5015d250 2023-12-16T21:58:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b3effd640 1 -- 172.21.15.28:0/4022570226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b480b9670 con 0x7f6b5015d250 2023-12-16T21:58:14.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b500b4fa0 con 0x7f6b5015d250 2023-12-16T21:58:14.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b500674d0 con 0x7f6b5015d250 2023-12-16T21:58:14.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b3effd640 1 -- 172.21.15.28:0/4022570226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6b480b97d0 con 0x7f6b5015d250 2023-12-16T21:58:14.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b3effd640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b30066470 0x7f6b30068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b3effd640 1 -- 172.21.15.28:0/4022570226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b4811f6c0 con 0x7f6b5015d250 2023-12-16T21:58:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4e59c640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b30066470 0x7f6b30068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.414+0000 7f6b4e59c640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b30066470 0x7f6b30068930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6b400011f0 tx=0x7f6b40009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:14.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.418+0000 7f6b3effd640 1 -- 172.21.15.28:0/4022570226 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6b480ecbe0 con 0x7f6b5015d250 2023-12-16T21:58:14.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.578+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b500b7140 con 0x7f6b5015d250 2023-12-16T21:58:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b3effd640 1 -- 172.21.15.28:0/4022570226 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f6b500b7140 con 0x7f6b5015d250 2023-12-16T21:58:14.584 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:14.584 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b30066470 msgr2=0x7f6b30068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b30066470 0x7f6b30068930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6b400011f0 tx=0x7f6b40009040 comp rx=0 tx=0).stop 2023-12-16T21:58:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5015d250 msgr2=0x7f6b500c6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5015d250 0x7f6b500c6ad0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6b4801a510 tx=0x7f6b48019680 comp rx=0 tx=0).stop 2023-12-16T21:58:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 shutdown_connections 2023-12-16T21:58:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b30066470 0x7f6b30068930 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b5015d250 0x7f6b500c6ad0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b5009a0e0 0x7f6b50157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 --2- 172.21.15.28:0/4022570226 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b5006b640 0x7f6b500b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 >> 172.21.15.28:0/4022570226 conn(0x7f6b500d0f30 msgr2=0x7f6b5006c390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 shutdown_connections 2023-12-16T21:58:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.582+0000 7f6b54efc640 1 -- 172.21.15.28:0/4022570226 wait complete. 2023-12-16T21:58:14.603 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:14.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:14.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61b48fc640 1 Processor -- start 2023-12-16T21:58:14.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61b48fc640 1 -- start start 2023-12-16T21:58:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61b48fc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b0158ca0 0x7f61b015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61b48fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f61b00cba10 0x7f61b015be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61b48fc640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b015b670 con 0x7f61b00b42d0 2023-12-16T21:58:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61b48fc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b015f410 con 0x7f61b00cba10 2023-12-16T21:58:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61b48fc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b015f6f0 con 0x7f61b0158ca0 2023-12-16T21:58:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61aed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f61b00b42d0 0x7f61b00703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43914/0 (socket says 172.21.15.28:43914) 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61aed9d640 1 -- 172.21.15.28:0/4182187646 learned_addr learned my addr 172.21.15.28:0/4182187646 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ae59c640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b0158ca0 0x7f61b015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61add9b640 1 --2- 172.21.15.28:0/4182187646 >> v2:172.21.15.122:3300/0 conn(0x7f61b00cba10 0x7f61b015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817988044 0 0) 0x7f61b015b670 con 0x7f61b00b42d0 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6198003530 con 0x7f61b00b42d0 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2876484342 0 0) 0x7f6198003530 con 0x7f61b00b42d0 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b0158ca0 msgr2=0x7f61b015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b0158ca0 0x7f61b015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 >> v2:172.21.15.122:3300/0 conn(0x7f61b00cba10 msgr2=0x7f61b015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 --2- 172.21.15.28:0/4182187646 >> v2:172.21.15.122:3300/0 conn(0x7f61b00cba10 0x7f61b015be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b015b8e0 con 0x7f61b00b42d0 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f61a8001a00 con 0x7f61b00b42d0 2023-12-16T21:58:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 >> v1:172.21.15.28:6789/0 conn(0x7f61b00b42d0 legacy=0x7f61b00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b0158ca0 0x7f6198003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198003f00 0x7f61980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198004d10 0x7f6198005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b015f410 con 0x7f6198003f00 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b015f6f0 con 0x7f6198004d10 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619800b990 con 0x7f61b0158ca0 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61add9b640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198003f00 0x7f61980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61aed9d640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198004d10 0x7f6198005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61ae59c640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b0158ca0 0x7f6198003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61add9b640 1 -- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b0158ca0 msgr2=0x7f6198003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61add9b640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b0158ca0 0x7f6198003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61add9b640 1 -- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198004d10 msgr2=0x7f6198005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61add9b640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198004d10 0x7f6198005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61add9b640 1 -- 172.21.15.28:0/4182187646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198003530 con 0x7f6198003f00 2023-12-16T21:58:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.726+0000 7f61aed9d640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198004d10 0x7f6198005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61add9b640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198003f00 0x7f61980047d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009ab0 tx=0x7f61a401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8796f537715b9a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a40a5660 con 0x7f6198003f00 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a40a57c0 con 0x7f6198003f00 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61ad59a640 1 -- 172.21.15.28:0/4182187646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a40a5b70 con 0x7f6198003f00 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61b48fc640 1 -- 172.21.15.28:0/4182187646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198003f00 msgr2=0x7f61980047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61b48fc640 1 --2- 172.21.15.28:0/4182187646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198003f00 0x7f61980047d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009ab0 tx=0x7f61a401ca10 comp rx=0 tx=0).stop 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61aed9d640 1 -- 172.21.15.28:0/4182187646 reap_dead start 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61b48fc640 1 -- 172.21.15.28:0/4182187646 shutdown_connections 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61b48fc640 1 -- 172.21.15.28:0/4182187646 >> 172.21.15.28:0/4182187646 conn(0x7f61b00b0cc0 msgr2=0x7f61b00b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61b48fc640 1 -- 172.21.15.28:0/4182187646 shutdown_connections 2023-12-16T21:58:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61b48fc640 1 -- 172.21.15.28:0/4182187646 wait complete. 2023-12-16T21:58:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61b48fc640 1 Processor -- start 2023-12-16T21:58:14.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.730+0000 7f61b48fc640 1 -- start start 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198003f00 0x7f61b013d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b00cc880 0x7f61b00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b01f4b70 0x7f61b01f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b00cd240 con 0x7f61b00cc880 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b013fb70 con 0x7f6198003f00 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b013fe50 con 0x7f61b01f4b70 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b00cc880 0x7f61b00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b00cc880 0x7f61b00ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60492/0 (socket says 172.21.15.28:60492) 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 -- 172.21.15.28:0/3010423612 learned_addr learned my addr 172.21.15.28:0/3010423612 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61ae59c640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198003f00 0x7f61b013d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61aed9d640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b01f4b70 0x7f61b01f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 -- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b01f4b70 msgr2=0x7f61b01f4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b01f4b70 0x7f61b01f4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 -- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198003f00 msgr2=0x7f61b013d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198003f00 0x7f61b013d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 -- 172.21.15.28:0/3010423612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b0140130 con 0x7f61b00cc880 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61add9b640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b00cc880 0x7f61b00ccd00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009ab0 tx=0x7f6198005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f619effd640 1 -- 172.21.15.28:0/3010423612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a40be070 con 0x7f61b00cc880 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f619effd640 1 -- 172.21.15.28:0/3010423612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a4004020 con 0x7f61b00cc880 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b0140410 con 0x7f61b00cc880 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f619effd640 1 -- 172.21.15.28:0/3010423612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4004390 con 0x7f61b00cc880 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b013d690 con 0x7f61b00cc880 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b00693d0 con 0x7f61b00cc880 2023-12-16T21:58:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f619effd640 1 -- 172.21.15.28:0/3010423612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f61a40b4040 con 0x7f61b00cc880 2023-12-16T21:58:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f619effd640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6190066470 0x7f6190068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f619effd640 1 -- 172.21.15.28:0/3010423612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61a411dad0 con 0x7f61b00cc880 2023-12-16T21:58:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.734+0000 7f61ae59c640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6190066470 0x7f6190068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.738+0000 7f61ae59c640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6190066470 0x7f6190068930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009a00 tx=0x7f61a0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.738+0000 7f619effd640 1 -- 172.21.15.28:0/3010423612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f61a40eb1b0 con 0x7f61b00cc880 2023-12-16T21:58:14.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61b015be90 con 0x7f61b00cc880 2023-12-16T21:58:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f619effd640 1 -- 172.21.15.28:0/3010423612 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61a40ee7d0 con 0x7f61b00cc880 2023-12-16T21:58:14.900 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6190066470 msgr2=0x7f6190068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6190066470 0x7f6190068930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009a00 tx=0x7f61a0008040 comp rx=0 tx=0).stop 2023-12-16T21:58:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b00cc880 msgr2=0x7f61b00ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b00cc880 0x7f61b00ccd00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009ab0 tx=0x7f6198005800 comp rx=0 tx=0).stop 2023-12-16T21:58:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 shutdown_connections 2023-12-16T21:58:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6190066470 0x7f6190068930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61b01f4b70 0x7f61b01f4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61b00cc880 0x7f61b00ccd00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 --2- 172.21.15.28:0/3010423612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6198003f00 0x7f61b013d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 >> 172.21.15.28:0/3010423612 conn(0x7f61b00b0cc0 msgr2=0x7f61b00afb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 shutdown_connections 2023-12-16T21:58:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:14.898+0000 7f61b48fc640 1 -- 172.21.15.28:0/3010423612 wait complete. 2023-12-16T21:58:14.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:15.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb7a81640 1 Processor -- start 2023-12-16T21:58:15.049 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb7a81640 1 -- start start 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb7a81640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb015adb0 0x7fefb013d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb7a81640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb013d2b0 con 0x7fefb0071d90 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb7a81640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb00cdd70 con 0x7fefb013f5f0 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb7a81640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb0140520 con 0x7fefb015adb0 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb581d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fefb0071d90 0x7fefb0072170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43926/0 (socket says 172.21.15.28:43926) 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb581d640 1 -- 172.21.15.28:0/1262686475 learned_addr learned my addr 172.21.15.28:0/1262686475 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb501c640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb015adb0 0x7fefb013d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703069034 0 0) 0x7fefb013d2b0 con 0x7fefb0071d90 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c005740 con 0x7fefb0071d90 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 343295102 0 0) 0x7fef9c005740 con 0x7fefb0071d90 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb015adb0 msgr2=0x7fefb013d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb015adb0 0x7fefb013d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 >> v1:172.21.15.122:6789/0 conn(0x7fefb013f5f0 legacy=0x7fefb00ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb013ff60 con 0x7fefb0071d90 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fefa8001680 con 0x7fefb0071d90 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 >> v1:172.21.15.28:6789/0 conn(0x7fefb0071d90 legacy=0x7fefb0072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef9c005bc0 0x7fef9c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0069e0 0x7fef9c006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb015adb0 0x7fef9c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb0140520 con 0x7fef9c0069e0 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c00dc50 con 0x7fef9c005bc0 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c00de10 con 0x7fefb015adb0 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb501c640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0069e0 0x7fef9c006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb581d640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef9c005bc0 0x7fef9c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.046+0000 7fefb601e640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb015adb0 0x7fef9c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.050+0000 7fefb501c640 1 -- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb015adb0 msgr2=0x7fef9c0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.050+0000 7fefb501c640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb015adb0 0x7fef9c0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.050+0000 7fefb501c640 1 -- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef9c005bc0 msgr2=0x7fef9c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.050+0000 7fefb501c640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef9c005bc0 0x7fef9c005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.050+0000 7fefb501c640 1 -- 172.21.15.28:0/1262686475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c005740 con 0x7fef9c0069e0 2023-12-16T21:58:15.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.050+0000 7fefb501c640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0069e0 0x7fef9c006e60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009ab0 tx=0x7fefa001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06c11972c4112fc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa00a6510 con 0x7fef9c0069e0 2023-12-16T21:58:15.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa00a6670 con 0x7fef9c0069e0 2023-12-16T21:58:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb481b640 1 -- 172.21.15.28:0/1262686475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa00a6a20 con 0x7fef9c0069e0 2023-12-16T21:58:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- 172.21.15.28:0/1262686475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0069e0 msgr2=0x7fef9c006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 --2- 172.21.15.28:0/1262686475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef9c0069e0 0x7fef9c006e60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009ab0 tx=0x7fefa001ca10 comp rx=0 tx=0).stop 2023-12-16T21:58:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb601e640 1 -- 172.21.15.28:0/1262686475 reap_dead start 2023-12-16T21:58:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- 172.21.15.28:0/1262686475 shutdown_connections 2023-12-16T21:58:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- 172.21.15.28:0/1262686475 >> 172.21.15.28:0/1262686475 conn(0x7fefb00b0cc0 msgr2=0x7fefb00b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- 172.21.15.28:0/1262686475 shutdown_connections 2023-12-16T21:58:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- 172.21.15.28:0/1262686475 wait complete. 2023-12-16T21:58:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 Processor -- start 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- start start 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefb0150370 0x7fefb01542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefb0151690 0x7fefb0152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb0152dc0 0x7fefb012ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb0153e50 con 0x7fefb0150370 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb0153240 con 0x7fefb0151690 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb0153520 con 0x7fefb0152dc0 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefb0150370 0x7fefb01542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefb0150370 0x7fefb01542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60520/0 (socket says 172.21.15.28:60520) 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 -- 172.21.15.28:0/419212984 learned_addr learned my addr 172.21.15.28:0/419212984 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb601e640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb0152dc0 0x7fefb012ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb501c640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefb0151690 0x7fefb0152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 -- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb0152dc0 msgr2=0x7fefb012ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb0152dc0 0x7fefb012ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 -- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefb0151690 msgr2=0x7fefb0152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefb0151690 0x7fefb0152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 -- 172.21.15.28:0/419212984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0153800 con 0x7fefb0150370 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb601e640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb0152dc0 0x7fefb012ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb501c640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefb0151690 0x7fefb0152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb581d640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefb0150370 0x7fefb01542e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fefa80017b0 tx=0x7fefa8012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefa67fc640 1 -- 172.21.15.28:0/419212984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8014810 con 0x7fefb0150370 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefa67fc640 1 -- 172.21.15.28:0/419212984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa8014970 con 0x7fefb0150370 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb0153ae0 con 0x7fefb0150370 2023-12-16T21:58:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefa67fc640 1 -- 172.21.15.28:0/419212984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa8014d00 con 0x7fefb0150370 2023-12-16T21:58:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.054+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb0154820 con 0x7fefb0150370 2023-12-16T21:58:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.058+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb00693d0 con 0x7fefb0150370 2023-12-16T21:58:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.058+0000 7fefa67fc640 1 -- 172.21.15.28:0/419212984 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fefa80153e0 con 0x7fefb0150370 2023-12-16T21:58:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.058+0000 7fefa67fc640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fef940664a0 0x7fef94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.058+0000 7fefa67fc640 1 -- 172.21.15.28:0/419212984 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefa8086b80 con 0x7fefb0150370 2023-12-16T21:58:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.058+0000 7fefb501c640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fef940664a0 0x7fef94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.058+0000 7fefb501c640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fef940664a0 0x7fef94068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fefa00af000 tx=0x7fefa0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.062+0000 7fefa67fc640 1 -- 172.21.15.28:0/419212984 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fefa8053f70 con 0x7fefb0150370 2023-12-16T21:58:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fefb00cc1e0 con 0x7fefb0150370 2023-12-16T21:58:15.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefa67fc640 1 -- 172.21.15.28:0/419212984 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fefa801c020 con 0x7fefb0150370 2023-12-16T21:58:15.221 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fef940664a0 msgr2=0x7fef94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fef940664a0 0x7fef94068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fefa00af000 tx=0x7fefa0008040 comp rx=0 tx=0).stop 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefb0150370 msgr2=0x7fefb01542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefb0150370 0x7fefb01542e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fefa80017b0 tx=0x7fefa8012920 comp rx=0 tx=0).stop 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 shutdown_connections 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fef940664a0 0x7fef94068960 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fefb0152dc0 0x7fefb012ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fefb0151690 0x7fefb0152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 --2- 172.21.15.28:0/419212984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefb0150370 0x7fefb01542e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 >> 172.21.15.28:0/419212984 conn(0x7fefb00b0cc0 msgr2=0x7fefb00b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 shutdown_connections 2023-12-16T21:58:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.218+0000 7fefb7a81640 1 -- 172.21.15.28:0/419212984 wait complete. 2023-12-16T21:58:15.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:15.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.358+0000 7f181cf78640 1 Processor -- start 2023-12-16T21:58:15.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.358+0000 7f181cf78640 1 -- start start 2023-12-16T21:58:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181cf78640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180ae830 0x7f181815d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181cf78640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f181815d710 0x7f18180ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181cf78640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18181575b0 con 0x7f18181571d0 2023-12-16T21:58:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181cf78640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181815dba0 con 0x7f181815d710 2023-12-16T21:58:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181cf78640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1818071390 con 0x7f18180ae830 2023-12-16T21:58:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181659c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f18181571d0 0x7f1818070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43940/0 (socket says 172.21.15.28:43940) 2023-12-16T21:58:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181659c640 1 -- 172.21.15.28:0/4234635032 learned_addr learned my addr 172.21.15.28:0/4234635032 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1815d9b640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180ae830 0x7f181815d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205545387 0 0) 0x7f18181575b0 con 0x7f18181571d0 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1816d9d640 1 --2- 172.21.15.28:0/4234635032 >> v2:172.21.15.122:3300/0 conn(0x7f181815d710 0x7f18180ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1804003530 con 0x7f18181571d0 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2163915108 0 0) 0x7f1804003530 con 0x7f18181571d0 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180ae830 msgr2=0x7f181815d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180ae830 0x7f181815d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 >> v2:172.21.15.122:3300/0 conn(0x7f181815d710 msgr2=0x7f18180ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 --2- 172.21.15.28:0/4234635032 >> v2:172.21.15.122:3300/0 conn(0x7f181815d710 0x7f18180ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818071600 con 0x7f18181571d0 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f1810001680 con 0x7f18181571d0 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 >> v1:172.21.15.28:6789/0 conn(0x7f18181571d0 legacy=0x7f1818070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1804003980 0x7f1804003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180ae830 0x7f18040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1804004d10 0x7f1804005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181815dba0 con 0x7f1804004d10 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1818071390 con 0x7f1804003980 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1804005960 con 0x7f18180ae830 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1816d9d640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1804004d10 0x7f1804005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f181659c640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1804003980 0x7f1804003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1815d9b640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180ae830 0x7f18040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1816d9d640 1 -- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180ae830 msgr2=0x7f18040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1816d9d640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180ae830 0x7f18040047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1816d9d640 1 -- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1804003980 msgr2=0x7f1804003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1816d9d640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1804003980 0x7f1804003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.362+0000 7f1816d9d640 1 -- 172.21.15.28:0/4234635032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1804003530 con 0x7f1804004d10 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1816d9d640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1804004d10 0x7f1804005190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1808009fc0 tx=0x7f180801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6466debda9b90a82 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18080a5cb0 con 0x7f1804004d10 2023-12-16T21:58:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180800f040 con 0x7f1804004d10 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181559a640 1 -- 172.21.15.28:0/4234635032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18080ad680 con 0x7f1804004d10 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- 172.21.15.28:0/4234635032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1804004d10 msgr2=0x7f1804005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 --2- 172.21.15.28:0/4234635032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1804004d10 0x7f1804005190 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1808009fc0 tx=0x7f180801cca0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1816d9d640 1 -- 172.21.15.28:0/4234635032 reap_dead start 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- 172.21.15.28:0/4234635032 shutdown_connections 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- 172.21.15.28:0/4234635032 >> 172.21.15.28:0/4234635032 conn(0x7f18180b0cc0 msgr2=0x7f18180b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- 172.21.15.28:0/4234635032 shutdown_connections 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- 172.21.15.28:0/4234635032 wait complete. 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 Processor -- start 2023-12-16T21:58:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- start start 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1818137dc0 0x7f18181f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18181f3450 0x7f18181575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180adc90 0x7f1818157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181815ad00 con 0x7f18181f3450 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18181381a0 con 0x7f1818137dc0 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181cf78640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181815afa0 con 0x7f18180adc90 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18181f3450 0x7f18181575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18181f3450 0x7f18181575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60542/0 (socket says 172.21.15.28:60542) 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 -- 172.21.15.28:0/952403817 learned_addr learned my addr 172.21.15.28:0/952403817 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181659c640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1818137dc0 0x7f18181f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1816d9d640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180adc90 0x7f1818157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 -- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180adc90 msgr2=0x7f1818157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180adc90 0x7f1818157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 -- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1818137dc0 msgr2=0x7f18181f2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1818137dc0 0x7f18181f2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 -- 172.21.15.28:0/952403817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181815b280 con 0x7f18181f3450 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f181659c640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1818137dc0 0x7f18181f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1816d9d640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180adc90 0x7f1818157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f1815d9b640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18181f3450 0x7f18181575b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f18000115e0 tx=0x7f1800011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.366+0000 7f180effd640 1 -- 172.21.15.28:0/952403817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000f8f0 con 0x7f18181f3450 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f180effd640 1 -- 172.21.15.28:0/952403817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180000fa50 con 0x7f18181f3450 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f180effd640 1 -- 172.21.15.28:0/952403817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000fdc0 con 0x7f18181f3450 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181815b560 con 0x7f18181f3450 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1818155500 con 0x7f18181f3450 2023-12-16T21:58:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18180693d0 con 0x7f18181f3450 2023-12-16T21:58:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f180effd640 1 -- 172.21.15.28:0/952403817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1800015460 con 0x7f18181f3450 2023-12-16T21:58:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f180effd640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f17f80664a0 0x7f17f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f180effd640 1 -- 172.21.15.28:0/952403817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18000843e0 con 0x7f18181f3450 2023-12-16T21:58:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f181659c640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f17f80664a0 0x7f17f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.370+0000 7f181659c640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f17f80664a0 0x7f17f8068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1810000c10 tx=0x7f1810009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.374+0000 7f180effd640 1 -- 172.21.15.28:0/952403817 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f18000517d0 con 0x7f18181f3450 2023-12-16T21:58:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.526+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1818156620 con 0x7f18181f3450 2023-12-16T21:58:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.526+0000 7f180effd640 1 -- 172.21.15.28:0/952403817 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1800054df0 con 0x7f18181f3450 2023-12-16T21:58:15.531 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f17f80664a0 msgr2=0x7f17f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f17f80664a0 0x7f17f8068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1810000c10 tx=0x7f1810009040 comp rx=0 tx=0).stop 2023-12-16T21:58:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18181f3450 msgr2=0x7f18181575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18181f3450 0x7f18181575b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f18000115e0 tx=0x7f1800011aa0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 shutdown_connections 2023-12-16T21:58:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f17f80664a0 0x7f17f8068960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f18180adc90 0x7f1818157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18181f3450 0x7f18181575b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 --2- 172.21.15.28:0/952403817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1818137dc0 0x7f18181f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 >> 172.21.15.28:0/952403817 conn(0x7f18180b0cc0 msgr2=0x7f18180af450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 shutdown_connections 2023-12-16T21:58:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.530+0000 7f181cf78640 1 -- 172.21.15.28:0/952403817 wait complete. 2023-12-16T21:58:15.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:58:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452e68d640 1 Processor -- start 2023-12-16T21:58:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452e68d640 1 -- start start 2023-12-16T21:58:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452e68d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4528140180 0x7f4528071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452e68d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4528071280 con 0x7f452806c770 2023-12-16T21:58:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452e68d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f452806cbe0 con 0x7f452806c2c0 2023-12-16T21:58:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452e68d640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45281408c0 con 0x7f4528140180 2023-12-16T21:58:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452cc2a640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f452806c770 0x7f4528070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43944/0 (socket says 172.21.15.28:43944) 2023-12-16T21:58:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452cc2a640 1 -- 172.21.15.28:0/1325754921 learned_addr learned my addr 172.21.15.28:0/1325754921 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4527fff640 1 --2- 172.21.15.28:0/1325754921 >> v2:172.21.15.122:3300/0 conn(0x7f4528140180 0x7f4528071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065517724 0 0) 0x7f4528071280 con 0x7f452806c770 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4514003530 con 0x7f452806c770 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127274113 0 0) 0x7f4514003530 con 0x7f452806c770 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 >> v2:172.21.15.122:3300/0 conn(0x7f4528140180 msgr2=0x7f4528071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 --2- 172.21.15.28:0/1325754921 >> v2:172.21.15.122:3300/0 conn(0x7f4528140180 0x7f4528071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 >> v1:172.21.15.122:6789/0 conn(0x7f452806c2c0 legacy=0x7f4528070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452815db80 con 0x7f452806c770 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f4518001a00 con 0x7f452806c770 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 >> v1:172.21.15.28:6789/0 conn(0x7f452806c770 legacy=0x7f4528070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4514003980 0x7f4514003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45140047d0 0x7f4514004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514005190 0x7f451400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45281408c0 con 0x7f4514005190 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451400c130 con 0x7f45140047d0 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451400c2f0 con 0x7f4514003980 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452cc2a640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514005190 0x7f451400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f45277fe640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45140047d0 0x7f4514004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f4527fff640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4514003980 0x7f4514003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452cc2a640 1 -- 172.21.15.28:0/1325754921 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4514003980 msgr2=0x7f4514003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452cc2a640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4514003980 0x7f4514003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452cc2a640 1 -- 172.21.15.28:0/1325754921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45140047d0 msgr2=0x7f4514004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452cc2a640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45140047d0 0x7f4514004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.658+0000 7f452cc2a640 1 -- 172.21.15.28:0/1325754921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4514003530 con 0x7f4514005190 2023-12-16T21:58:15.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452cc2a640 1 -- 172.21.15.28:0/1325754921 reap_dead start 2023-12-16T21:58:15.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452cc2a640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514005190 0x7f451400b990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f451800a830 tx=0x7f451801faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32440a2ad8dfa84a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45180aa6d0 con 0x7f4514005190 2023-12-16T21:58:15.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45180aa850 con 0x7f4514005190 2023-12-16T21:58:15.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f4526ffd640 1 -- 172.21.15.28:0/1325754921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45180aac00 con 0x7f4514005190 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- 172.21.15.28:0/1325754921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514005190 msgr2=0x7f451400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514005190 0x7f451400b990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f451800a830 tx=0x7f451801faa0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- 172.21.15.28:0/1325754921 shutdown_connections 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 --2- 172.21.15.28:0/1325754921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4514005190 0x7f451400b990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- 172.21.15.28:0/1325754921 >> 172.21.15.28:0/1325754921 conn(0x7f452806a530 msgr2=0x7f452806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- 172.21.15.28:0/1325754921 shutdown_connections 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- 172.21.15.28:0/1325754921 wait complete. 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 Processor -- start 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- start start 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4514005190 0x7f45280ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452815bee0 0x7f452815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f45280af8a0 0x7f45280afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45280b0260 con 0x7f452815bee0 2023-12-16T21:58:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45281308d0 con 0x7f4514005190 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452e68d640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4528130b40 con 0x7f45280af8a0 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452815bee0 0x7f452815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452815bee0 0x7f452815c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60554/0 (socket says 172.21.15.28:60554) 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 -- 172.21.15.28:0/4127887226 learned_addr learned my addr 172.21.15.28:0/4127887226 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f452cc2a640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f45280af8a0 0x7f45280afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f4527fff640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4514005190 0x7f45280ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 -- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f45280af8a0 msgr2=0x7f45280afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f45280af8a0 0x7f45280afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 -- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4514005190 msgr2=0x7f45280ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4514005190 0x7f45280ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 -- 172.21.15.28:0/4127887226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4528130e20 con 0x7f452815bee0 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.662+0000 7f45277fe640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452815bee0 0x7f452815c360 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f451000e3f0 tx=0x7f451000e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f4524ff9640 1 -- 172.21.15.28:0/4127887226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45100052e0 con 0x7f452815bee0 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f4524ff9640 1 -- 172.21.15.28:0/4127887226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4510005440 con 0x7f452815bee0 2023-12-16T21:58:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4528131100 con 0x7f452815bee0 2023-12-16T21:58:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f4524ff9640 1 -- 172.21.15.28:0/4127887226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510003820 con 0x7f452815bee0 2023-12-16T21:58:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45280ce7e0 con 0x7f452815bee0 2023-12-16T21:58:15.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4528069440 con 0x7f452815bee0 2023-12-16T21:58:15.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f4524ff9640 1 -- 172.21.15.28:0/4127887226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4510003980 con 0x7f452815bee0 2023-12-16T21:58:15.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f4524ff9640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f45080664a0 0x7f4508068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:15.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f4524ff9640 1 -- 172.21.15.28:0/4127887226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4510083510 con 0x7f452815bee0 2023-12-16T21:58:15.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f4527fff640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f45080664a0 0x7f4508068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:15.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.666+0000 7f4527fff640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f45080664a0 0x7f4508068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f45200099b0 tx=0x7f4520008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.670+0000 7f4524ff9640 1 -- 172.21.15.28:0/4127887226 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4510050ad0 con 0x7f452815bee0 2023-12-16T21:58:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.838+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f45280b0880 con 0x7f452815bee0 2023-12-16T21:58:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.838+0000 7f4524ff9640 1 -- 172.21.15.28:0/4127887226 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4510081e60 con 0x7f452815bee0 2023-12-16T21:58:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2023-12-16T21:58:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.838+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f45080664a0 msgr2=0x7f4508068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.838+0000 7f452e68d640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f45080664a0 0x7f4508068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f45200099b0 tx=0x7f4520008040 comp rx=0 tx=0).stop 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.838+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452815bee0 msgr2=0x7f452815c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.838+0000 7f452e68d640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452815bee0 0x7f452815c360 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f451000e3f0 tx=0x7f451000e8b0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.842+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 shutdown_connections 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.842+0000 7f452e68d640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f45080664a0 0x7f4508068960 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.842+0000 7f452e68d640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f45280af8a0 0x7f45280afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.842+0000 7f452e68d640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452815bee0 0x7f452815c360 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.842+0000 7f452e68d640 1 --2- 172.21.15.28:0/4127887226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4514005190 0x7f45280ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.842+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 >> 172.21.15.28:0/4127887226 conn(0x7f452806a530 msgr2=0x7f45280b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.842+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 shutdown_connections 2023-12-16T21:58:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:15.842+0000 7f452e68d640 1 -- 172.21.15.28:0/4127887226 wait complete. 2023-12-16T21:58:15.860 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:58:15.860 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:15.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:16.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe496bd8640 1 Processor -- start 2023-12-16T21:58:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe496bd8640 1 -- start start 2023-12-16T21:58:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe496bd8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 0x7fe49006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe496bd8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe490072180 0x7fe49006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe496bd8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4900710e0 con 0x7fe490070460 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe496bd8640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49015f010 con 0x7fe490072180 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe496bd8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900713c0 con 0x7fe490144840 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe495175640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe490070460 0x7fe4900708e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43946/0 (socket says 172.21.15.28:43946) 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe495175640 1 -- 172.21.15.28:0/1505732418 learned_addr learned my addr 172.21.15.28:0/1505732418 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe494974640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 0x7fe49006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48ffff640 1 --2- 172.21.15.28:0/1505732418 >> v2:172.21.15.122:3300/0 conn(0x7fe490072180 0x7fe49006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846511416 0 0) 0x7fe4900710e0 con 0x7fe490070460 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe480003530 con 0x7fe490070460 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3855405520 0 0) 0x7fe480003530 con 0x7fe490070460 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 msgr2=0x7fe49006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 0x7fe49006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 >> v2:172.21.15.122:3300/0 conn(0x7fe490072180 msgr2=0x7fe49006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 --2- 172.21.15.28:0/1505732418 >> v2:172.21.15.122:3300/0 conn(0x7fe490072180 0x7fe49006c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4900716a0 con 0x7fe490070460 2023-12-16T21:58:16.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fe488001a00 con 0x7fe490070460 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 >> v1:172.21.15.28:6789/0 conn(0x7fe490070460 legacy=0x7fe4900708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480003980 0x7fe480003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 0x7fe4800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe480004d10 0x7fe480005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900713c0 con 0x7fe480003980 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48000b990 con 0x7fe480004d10 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48000bad0 con 0x7fe490144840 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe494974640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480003980 0x7fe480003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe495175640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe480004d10 0x7fe480005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48ffff640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 0x7fe4800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe494974640 1 -- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 msgr2=0x7fe4800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe494974640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 0x7fe4800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe494974640 1 -- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe480004d10 msgr2=0x7fe480005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe494974640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe480004d10 0x7fe480005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe494974640 1 -- 172.21.15.28:0/1505732418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe480003530 con 0x7fe480003980 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe495175640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe480004d10 0x7fe480005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.022+0000 7fe48ffff640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe490144840 0x7fe4800047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe494974640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480003980 0x7fe480003da0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe484009b10 tx=0x7fe48401ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eaa9440a72d857e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840a6510 con 0x7fe480003980 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4840a6670 con 0x7fe480003980 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe48f7fe640 1 -- 172.21.15.28:0/1505732418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840a6a20 con 0x7fe480003980 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- 172.21.15.28:0/1505732418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480003980 msgr2=0x7fe480003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 --2- 172.21.15.28:0/1505732418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480003980 0x7fe480003da0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe484009b10 tx=0x7fe48401ca20 comp rx=0 tx=0).stop 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe495175640 1 -- 172.21.15.28:0/1505732418 reap_dead start 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- 172.21.15.28:0/1505732418 shutdown_connections 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- 172.21.15.28:0/1505732418 >> 172.21.15.28:0/1505732418 conn(0x7fe49006a530 msgr2=0x7fe49006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- 172.21.15.28:0/1505732418 shutdown_connections 2023-12-16T21:58:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- 172.21.15.28:0/1505732418 wait complete. 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 Processor -- start 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- start start 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe49015a930 0x7fe4900a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4900a96e0 0x7fe4900a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4900a8b30 0x7fe4901f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900a9b60 con 0x7fe4900a96e0 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900a8fb0 con 0x7fe4900a8b30 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe496bd8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900ca650 con 0x7fe49015a930 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4900a96e0 0x7fe4900a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe48ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4900a96e0 0x7fe4900a85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60578/0 (socket says 172.21.15.28:60578) 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.026+0000 7fe48ffff640 1 -- 172.21.15.28:0/1091601631 learned_addr learned my addr 172.21.15.28:0/1091601631 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:16.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48ffff640 1 -- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe49015a930 msgr2=0x7fe4900a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48ffff640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe49015a930 0x7fe4900a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48ffff640 1 -- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4900a8b30 msgr2=0x7fe4901f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48ffff640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4900a8b30 0x7fe4901f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48ffff640 1 -- 172.21.15.28:0/1091601631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4900ca880 con 0x7fe4900a96e0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48ffff640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4900a96e0 0x7fe4900a85f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009f00 tx=0x7fe47c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48d7fa640 1 -- 172.21.15.28:0/1091601631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00cae0 con 0x7fe4900a96e0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48d7fa640 1 -- 172.21.15.28:0/1091601631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c00cc40 con 0x7fe4900a96e0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4900cab60 con 0x7fe4900a96e0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48d7fa640 1 -- 172.21.15.28:0/1091601631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00ee60 con 0x7fe4900a96e0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4900cb120 con 0x7fe4900a96e0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe490069440 con 0x7fe4900a96e0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48d7fa640 1 -- 172.21.15.28:0/1091601631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe47c0135f0 con 0x7fe4900a96e0 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48d7fa640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe474066470 0x7fe474068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe48d7fa640 1 -- 172.21.15.28:0/1091601631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe47c0878c0 con 0x7fe4900a96e0 2023-12-16T21:58:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe494974640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe474066470 0x7fe474068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.030+0000 7fe494974640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe474066470 0x7fe474068930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe4840af000 tx=0x7fe484004740 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.034+0000 7fe48d7fa640 1 -- 172.21.15.28:0/1091601631 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe47c054290 con 0x7fe4900a96e0 2023-12-16T21:58:16.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.182+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe49006d170 con 0x7fe4900a96e0 2023-12-16T21:58:16.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe48d7fa640 1 -- 172.21.15.28:0/1091601631 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fe47c020020 con 0x7fe4900a96e0 2023-12-16T21:58:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:16.189 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:16.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe474066470 msgr2=0x7fe474068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe474066470 0x7fe474068930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe4840af000 tx=0x7fe484004740 comp rx=0 tx=0).stop 2023-12-16T21:58:16.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4900a96e0 msgr2=0x7fe4900a85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.191 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4900a96e0 0x7fe4900a85f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009f00 tx=0x7fe47c00ec10 comp rx=0 tx=0).stop 2023-12-16T21:58:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 shutdown_connections 2023-12-16T21:58:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe474066470 0x7fe474068930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4900a8b30 0x7fe4901f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4900a96e0 0x7fe4900a85f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 --2- 172.21.15.28:0/1091601631 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe49015a930 0x7fe4900a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 >> 172.21.15.28:0/1091601631 conn(0x7fe49006a530 msgr2=0x7fe49015cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.186+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 shutdown_connections 2023-12-16T21:58:16.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.190+0000 7fe496bd8640 1 -- 172.21.15.28:0/1091601631 wait complete. 2023-12-16T21:58:16.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:16.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f3835c49640 1 Processor -- start 2023-12-16T21:58:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f3835c49640 1 -- start start 2023-12-16T21:58:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f3835c49640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3830071d90 0x7f3830072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f3835c49640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 0x7f383013d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f3835c49640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383013d2b0 con 0x7f383013f5f0 2023-12-16T21:58:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f3835c49640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3830140520 con 0x7f3830071d90 2023-12-16T21:58:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f3835c49640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38300cd7f0 con 0x7f383015adb0 2023-12-16T21:58:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f383013f5f0 0x7f38300ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43948/0 (socket says 172.21.15.28:43948) 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382ffff640 1 -- 172.21.15.28:0/1566676189 learned_addr learned my addr 172.21.15.28:0/1566676189 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382f7fe640 1 --2- 172.21.15.28:0/1566676189 >> v2:172.21.15.28:3300/0 conn(0x7f3830071d90 0x7f3830072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382effd640 1 --2- 172.21.15.28:0/1566676189 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 0x7f383013d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382e7fc640 1 -- 172.21.15.28:0/1566676189 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967235444 0 0) 0x7f383013d2b0 con 0x7f383013f5f0 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382e7fc640 1 -- 172.21.15.28:0/1566676189 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3818003530 con 0x7f383013f5f0 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382f7fe640 1 -- 172.21.15.28:0/1566676189 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 msgr2=0x7f383013d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382f7fe640 1 --2- 172.21.15.28:0/1566676189 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 0x7f383013d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382f7fe640 1 -- 172.21.15.28:0/1566676189 >> v1:172.21.15.28:6789/0 conn(0x7f383013f5f0 legacy=0x7f38300ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382e7fc640 1 -- 172.21.15.28:0/1566676189 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543593181 0 0) 0x7f3818003530 con 0x7f383013f5f0 2023-12-16T21:58:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.334+0000 7f382f7fe640 1 -- 172.21.15.28:0/1566676189 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300cdd70 con 0x7f3830071d90 2023-12-16T21:58:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f382f7fe640 1 --2- 172.21.15.28:0/1566676189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3830071d90 0x7f3830072170 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f38200097c0 tx=0x7f382001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144beb56b75fe29f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f382e7fc640 1 -- 172.21.15.28:0/1566676189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828001a00 con 0x7f3830071d90 2023-12-16T21:58:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f382e7fc640 1 -- 172.21.15.28:0/1566676189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38200a7050 con 0x7f3830071d90 2023-12-16T21:58:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f382e7fc640 1 -- 172.21.15.28:0/1566676189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38200ab620 con 0x7f3830071d90 2023-12-16T21:58:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 -- 172.21.15.28:0/1566676189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3830071d90 msgr2=0x7f3830072170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 --2- 172.21.15.28:0/1566676189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3830071d90 0x7f3830072170 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f38200097c0 tx=0x7f382001c3a0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 -- 172.21.15.28:0/1566676189 shutdown_connections 2023-12-16T21:58:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 --2- 172.21.15.28:0/1566676189 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 0x7f383013d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 --2- 172.21.15.28:0/1566676189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3830071d90 0x7f3830072170 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 -- 172.21.15.28:0/1566676189 >> 172.21.15.28:0/1566676189 conn(0x7f38300b0cc0 msgr2=0x7f38300b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 -- 172.21.15.28:0/1566676189 shutdown_connections 2023-12-16T21:58:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 -- 172.21.15.28:0/1566676189 wait complete. 2023-12-16T21:58:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 Processor -- start 2023-12-16T21:58:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.338+0000 7f3835c49640 1 -- start start 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3830071d90 0x7f3830150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383013f5f0 0x7f3830070c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 0x7f38300711c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38300adcb0 con 0x7f383013f5f0 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3830134a40 con 0x7f3830071d90 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38300ae140 con 0x7f383015adb0 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383013f5f0 0x7f3830070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383013f5f0 0x7f3830070c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60588/0 (socket says 172.21.15.28:60588) 2023-12-16T21:58:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 -- 172.21.15.28:0/591405229 learned_addr learned my addr 172.21.15.28:0/591405229 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382ffff640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 0x7f38300711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382f7fe640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3830071d90 0x7f3830150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 -- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 msgr2=0x7f38300711c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 0x7f38300711c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 -- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3830071d90 msgr2=0x7f3830150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3830071d90 0x7f3830150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 -- 172.21.15.28:0/591405229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3820009420 con 0x7f383013f5f0 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382effd640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383013f5f0 0x7f3830070c80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f382400e870 tx=0x7f382400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3817fff640 1 -- 172.21.15.28:0/591405229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824009510 con 0x7f383013f5f0 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3817fff640 1 -- 172.21.15.28:0/591405229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382400eec0 con 0x7f383013f5f0 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3817fff640 1 -- 172.21.15.28:0/591405229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382400cb40 con 0x7f383013f5f0 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3830150b50 con 0x7f383013f5f0 2023-12-16T21:58:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38300ad660 con 0x7f383013f5f0 2023-12-16T21:58:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38300693d0 con 0x7f383013f5f0 2023-12-16T21:58:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3817fff640 1 -- 172.21.15.28:0/591405229 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f382400cca0 con 0x7f383013f5f0 2023-12-16T21:58:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3817fff640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f381c0664a0 0x7f381c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f3817fff640 1 -- 172.21.15.28:0/591405229 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3824083f40 con 0x7f383013f5f0 2023-12-16T21:58:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.342+0000 7f382f7fe640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f381c0664a0 0x7f381c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.346+0000 7f382f7fe640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f381c0664a0 0x7f381c068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f38200ad000 tx=0x7f38200a6000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.346+0000 7f3817fff640 1 -- 172.21.15.28:0/591405229 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3824051450 con 0x7f383013f5f0 2023-12-16T21:58:16.507 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.502+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f383006e710 con 0x7f383013f5f0 2023-12-16T21:58:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3817fff640 1 -- 172.21.15.28:0/591405229 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3824017020 con 0x7f383013f5f0 2023-12-16T21:58:16.508 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f381c0664a0 msgr2=0x7f381c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f381c0664a0 0x7f381c068960 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f38200ad000 tx=0x7f38200a6000 comp rx=0 tx=0).stop 2023-12-16T21:58:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383013f5f0 msgr2=0x7f3830070c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383013f5f0 0x7f3830070c80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f382400e870 tx=0x7f382400ec70 comp rx=0 tx=0).stop 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 shutdown_connections 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f381c0664a0 0x7f381c068960 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f383015adb0 0x7f38300711c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383013f5f0 0x7f3830070c80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 --2- 172.21.15.28:0/591405229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3830071d90 0x7f3830150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 >> 172.21.15.28:0/591405229 conn(0x7f38300b0cc0 msgr2=0x7f38300afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 shutdown_connections 2023-12-16T21:58:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.506+0000 7f3835c49640 1 -- 172.21.15.28:0/591405229 wait complete. 2023-12-16T21:58:16.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.650+0000 7fb07ff40640 1 Processor -- start 2023-12-16T21:58:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.650+0000 7fb07ff40640 1 -- start start 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.650+0000 7fb07ff40640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb07815c140 0x7fb0780cf700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.650+0000 7fb07ff40640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb0780cfc40 0x7fb0780d1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.650+0000 7fb07ff40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07815f380 con 0x7fb078144c10 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.650+0000 7fb07ff40640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb07815f660 con 0x7fb0780cfc40 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.650+0000 7fb07ff40640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0780b4510 con 0x7fb07815c140 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.650+0000 7fb07e4dd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb0780cfc40 0x7fb0780d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07d4db640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb07815c140 0x7fb0780cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07e4dd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb0780cfc40 0x7fb0780d1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:47166/0 (socket says 172.21.15.28:47166) 2023-12-16T21:58:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07e4dd640 1 -- 172.21.15.28:0/1271539798 learned_addr learned my addr 172.21.15.28:0/1271539798 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07dcdc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fb078144c10 0x7fb07815eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:56082/0 (socket says 172.21.15.28:56082) 2023-12-16T21:58:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07d4db640 1 -- 172.21.15.28:0/1271539798 >> v2:172.21.15.122:3300/0 conn(0x7fb0780cfc40 msgr2=0x7fb0780d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07d4db640 1 --2- 172.21.15.28:0/1271539798 >> v2:172.21.15.122:3300/0 conn(0x7fb0780cfc40 0x7fb0780d1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07d4db640 1 -- 172.21.15.28:0/1271539798 >> v1:172.21.15.122:6789/0 conn(0x7fb078144c10 legacy=0x7fb07815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07d4db640 1 -- 172.21.15.28:0/1271539798 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0780b47f0 con 0x7fb07815c140 2023-12-16T21:58:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07d4db640 1 --2- 172.21.15.28:0/1271539798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb07815c140 0x7fb0780cf700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb0680097c0 tx=0x7fb06801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2b10f97a22cc2f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ccda640 1 -- 172.21.15.28:0/1271539798 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0680ab600 con 0x7fb07815c140 2023-12-16T21:58:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ccda640 1 -- 172.21.15.28:0/1271539798 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0680a7050 con 0x7fb07815c140 2023-12-16T21:58:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ccda640 1 -- 172.21.15.28:0/1271539798 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0680ab980 con 0x7fb07815c140 2023-12-16T21:58:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ff40640 1 -- 172.21.15.28:0/1271539798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb07815c140 msgr2=0x7fb0780cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ff40640 1 --2- 172.21.15.28:0/1271539798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb07815c140 0x7fb0780cf700 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb0680097c0 tx=0x7fb06801c3c0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ff40640 1 -- 172.21.15.28:0/1271539798 shutdown_connections 2023-12-16T21:58:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ff40640 1 --2- 172.21.15.28:0/1271539798 >> v2:172.21.15.122:3300/0 conn(0x7fb0780cfc40 0x7fb0780d1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ff40640 1 --2- 172.21.15.28:0/1271539798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb07815c140 0x7fb0780cf700 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ff40640 1 -- 172.21.15.28:0/1271539798 >> 172.21.15.28:0/1271539798 conn(0x7fb07806a930 msgr2=0x7fb07806ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ff40640 1 -- 172.21.15.28:0/1271539798 shutdown_connections 2023-12-16T21:58:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.654+0000 7fb07ff40640 1 -- 172.21.15.28:0/1271539798 wait complete. 2023-12-16T21:58:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 Processor -- start 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 -- start start 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0780cfc40 0x7fb07812e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb078144c10 0x7fb07815b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815c140 0x7fb07812fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07815bab0 con 0x7fb07815c140 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07812f710 con 0x7fb0780cfc40 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07812eb00 con 0x7fb078144c10 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815c140 0x7fb07812fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815c140 0x7fb07812fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60592/0 (socket says 172.21.15.28:60592) 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 -- 172.21.15.28:0/516226771 learned_addr learned my addr 172.21.15.28:0/516226771 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07d4db640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb078144c10 0x7fb07815b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07dcdc640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0780cfc40 0x7fb07812e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 -- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb078144c10 msgr2=0x7fb07815b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb078144c10 0x7fb07815b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 -- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0780cfc40 msgr2=0x7fb07812e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0780cfc40 0x7fb07812e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 -- 172.21.15.28:0/516226771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb068009420 con 0x7fb07815c140 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07dcdc640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0780cfc40 0x7fb07812e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07e4dd640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815c140 0x7fb07812fba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb07400e7f0 tx=0x7fb07400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb06e7fc640 1 -- 172.21.15.28:0/516226771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07400eea0 con 0x7fb07815c140 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb06e7fc640 1 -- 172.21.15.28:0/516226771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07400c940 con 0x7fb07815c140 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07812ede0 con 0x7fb07815c140 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb06e7fc640 1 -- 172.21.15.28:0/516226771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07400ccd0 con 0x7fb07815c140 2023-12-16T21:58:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07812f3a0 con 0x7fb07815c140 2023-12-16T21:58:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.658+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07812f880 con 0x7fb07815c140 2023-12-16T21:58:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.662+0000 7fb06e7fc640 1 -- 172.21.15.28:0/516226771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fb074013470 con 0x7fb07815c140 2023-12-16T21:58:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.662+0000 7fb06e7fc640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb05c066470 0x7fb05c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.662+0000 7fb06e7fc640 1 -- 172.21.15.28:0/516226771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb074083fb0 con 0x7fb07815c140 2023-12-16T21:58:16.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.662+0000 7fb07dcdc640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb05c066470 0x7fb05c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.662+0000 7fb07dcdc640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb05c066470 0x7fb05c068930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0700012f0 tx=0x7fb070009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.666+0000 7fb06e7fc640 1 -- 172.21.15.28:0/516226771 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fb074051390 con 0x7fb07815c140 2023-12-16T21:58:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.822+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb078141910 con 0x7fb07815c140 2023-12-16T21:58:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.822+0000 7fb06e7fc640 1 -- 172.21.15.28:0/516226771 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0740549b0 con 0x7fb07815c140 2023-12-16T21:58:16.825 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb05c066470 msgr2=0x7fb05c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb05c066470 0x7fb05c068930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0700012f0 tx=0x7fb070009040 comp rx=0 tx=0).stop 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815c140 msgr2=0x7fb07812fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815c140 0x7fb07812fba0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb07400e7f0 tx=0x7fb07400ecb0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 shutdown_connections 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb05c066470 0x7fb05c068930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb07815c140 0x7fb07812fba0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb078144c10 0x7fb07815b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 --2- 172.21.15.28:0/516226771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0780cfc40 0x7fb07812e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 >> 172.21.15.28:0/516226771 conn(0x7fb07806a930 msgr2=0x7fb07815e050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 shutdown_connections 2023-12-16T21:58:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.826+0000 7fb07ff40640 1 -- 172.21.15.28:0/516226771 wait complete. 2023-12-16T21:58:16.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:16.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490c482640 1 Processor -- start 2023-12-16T21:58:16.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490c482640 1 -- start start 2023-12-16T21:58:16.973 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490c482640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f490406c500 0x7f490406c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490c482640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f49040b0ae0 0x7f49040b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490c482640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49040b1450 con 0x7f4904151fe0 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490c482640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490406d580 con 0x7f49040b0ae0 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490c482640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49041523c0 con 0x7f490406c500 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490aa1f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f49040b0ae0 0x7f49040b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490aa1f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f49040b0ae0 0x7f49040b0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60596/0 (socket says 172.21.15.28:60596) 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490aa1f640 1 -- 172.21.15.28:0/154398558 learned_addr learned my addr 172.21.15.28:0/154398558 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f4909a1d640 1 --2- 172.21.15.28:0/154398558 >> v2:172.21.15.122:3300/0 conn(0x7f490406c500 0x7f490406c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490aa1f640 1 -- 172.21.15.28:0/154398558 >> v2:172.21.15.122:3300/0 conn(0x7f490406c500 msgr2=0x7f490406c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490aa1f640 1 --2- 172.21.15.28:0/154398558 >> v2:172.21.15.122:3300/0 conn(0x7f490406c500 0x7f490406c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490aa1f640 1 -- 172.21.15.28:0/154398558 >> v1:172.21.15.122:6789/0 conn(0x7f4904151fe0 legacy=0x7f4904070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.970+0000 7f490aa1f640 1 -- 172.21.15.28:0/154398558 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490406cfc0 con 0x7f49040b0ae0 2023-12-16T21:58:16.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490aa1f640 1 --2- 172.21.15.28:0/154398558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f49040b0f10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f49040cc5f0 tx=0x7f48fc01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca833cffb229c092 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490921c640 1 -- 172.21.15.28:0/154398558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc0a5860 con 0x7f49040b0ae0 2023-12-16T21:58:16.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490921c640 1 -- 172.21.15.28:0/154398558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc0a59c0 con 0x7f49040b0ae0 2023-12-16T21:58:16.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490921c640 1 -- 172.21.15.28:0/154398558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc0a5d50 con 0x7f49040b0ae0 2023-12-16T21:58:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 -- 172.21.15.28:0/154398558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 msgr2=0x7f49040b0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 --2- 172.21.15.28:0/154398558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f49040b0f10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f49040cc5f0 tx=0x7f48fc01c9a0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 -- 172.21.15.28:0/154398558 shutdown_connections 2023-12-16T21:58:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 --2- 172.21.15.28:0/154398558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f49040b0f10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 --2- 172.21.15.28:0/154398558 >> v2:172.21.15.122:3300/0 conn(0x7f490406c500 0x7f490406c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 -- 172.21.15.28:0/154398558 >> 172.21.15.28:0/154398558 conn(0x7f490406a530 msgr2=0x7f490406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 -- 172.21.15.28:0/154398558 shutdown_connections 2023-12-16T21:58:16.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 -- 172.21.15.28:0/154398558 wait complete. 2023-12-16T21:58:16.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 Processor -- start 2023-12-16T21:58:16.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.974+0000 7f490c482640 1 -- start start 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f490406c500 0x7f4904144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f4904154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4904151fe0 0x7f4904155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490414c9c0 con 0x7f49040b0ae0 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490414c530 con 0x7f490406c500 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490414bc00 con 0x7f4904151fe0 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f4904154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f4904154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60608/0 (socket says 172.21.15.28:60608) 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 -- 172.21.15.28:0/3404972795 learned_addr learned my addr 172.21.15.28:0/3404972795 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490aa1f640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4904151fe0 0x7f4904155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490a21e640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f490406c500 0x7f4904144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 -- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4904151fe0 msgr2=0x7f4904155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4904151fe0 0x7f4904155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 -- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f490406c500 msgr2=0x7f4904144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f490406c500 0x7f4904144f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 -- 172.21.15.28:0/3404972795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc009b30 con 0x7f49040b0ae0 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490a21e640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f490406c500 0x7f4904144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490aa1f640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4904151fe0 0x7f4904155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f4909a1d640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f4904154b40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f48f400e930 tx=0x7f48f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f48faffd640 1 -- 172.21.15.28:0/3404972795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4009690 con 0x7f49040b0ae0 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f48faffd640 1 -- 172.21.15.28:0/3404972795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f400ca00 con 0x7f49040b0ae0 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490414bee0 con 0x7f49040b0ae0 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f48faffd640 1 -- 172.21.15.28:0/3404972795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f400cd70 con 0x7f49040b0ae0 2023-12-16T21:58:16.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490414e9d0 con 0x7f49040b0ae0 2023-12-16T21:58:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490414c6a0 con 0x7f49040b0ae0 2023-12-16T21:58:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f48faffd640 1 -- 172.21.15.28:0/3404972795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f48f4013470 con 0x7f49040b0ae0 2023-12-16T21:58:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f48faffd640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f48e80664a0 0x7f48e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f48faffd640 1 -- 172.21.15.28:0/3404972795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48f4084060 con 0x7f49040b0ae0 2023-12-16T21:58:16.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.978+0000 7f490a21e640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f48e80664a0 0x7f48e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.982+0000 7f490a21e640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f48e80664a0 0x7f48e8068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4900000f70 tx=0x7f4900009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:16.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:16.982+0000 7f48faffd640 1 -- 172.21.15.28:0/3404972795 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f48f40515a0 con 0x7f49040b0ae0 2023-12-16T21:58:17.141 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.138+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49040693d0 con 0x7f49040b0ae0 2023-12-16T21:58:17.141 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.138+0000 7f48faffd640 1 -- 172.21.15.28:0/3404972795 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48f4054bc0 con 0x7f49040b0ae0 2023-12-16T21:58:17.142 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f48e80664a0 msgr2=0x7f48e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f48e80664a0 0x7f48e8068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4900000f70 tx=0x7f4900009040 comp rx=0 tx=0).stop 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 msgr2=0x7f4904154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f4904154b40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f48f400e930 tx=0x7f48f400edf0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 shutdown_connections 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f48e80664a0 0x7f48e8068960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4904151fe0 0x7f4904155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49040b0ae0 0x7f4904154b40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 --2- 172.21.15.28:0/3404972795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f490406c500 0x7f4904144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 >> 172.21.15.28:0/3404972795 conn(0x7f490406a530 msgr2=0x7f490406a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 shutdown_connections 2023-12-16T21:58:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.142+0000 7f490c482640 1 -- 172.21.15.28:0/3404972795 wait complete. 2023-12-16T21:58:17.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:58:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5192b9640 1 Processor -- start 2023-12-16T21:58:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5192b9640 1 -- start start 2023-12-16T21:58:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5192b9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb51406c9d0 0x7fb514070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5192b9640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5140710e0 con 0x7fb514072180 2023-12-16T21:58:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5192b9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5140713c0 con 0x7fb514144840 2023-12-16T21:58:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5192b9640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51415f010 con 0x7fb51406c9d0 2023-12-16T21:58:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5137fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb514072180 0x7fb51415b610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43964/0 (socket says 172.21.15.28:43964) 2023-12-16T21:58:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5137fe640 1 -- 172.21.15.28:0/2906653145 learned_addr learned my addr 172.21.15.28:0/2906653145 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5127fc640 1 --2- 172.21.15.28:0/2906653145 >> v2:172.21.15.28:3300/0 conn(0x7fb51406c9d0 0x7fb514070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5127fc640 1 -- 172.21.15.28:0/2906653145 >> v1:172.21.15.122:6789/0 conn(0x7fb514144840 legacy=0x7fb51406c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5127fc640 1 -- 172.21.15.28:0/2906653145 >> v1:172.21.15.28:6789/0 conn(0x7fb514072180 legacy=0x7fb51415b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.278+0000 7fb5127fc640 1 -- 172.21.15.28:0/2906653145 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5140716a0 con 0x7fb51406c9d0 2023-12-16T21:58:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 --2- 172.21.15.28:0/2906653145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51406c9d0 0x7fb514070460 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0097c0 tx=0x7fb4fc01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb9467fd2a345af server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb511ffb640 1 -- 172.21.15.28:0/2906653145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc0ab610 con 0x7fb51406c9d0 2023-12-16T21:58:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb511ffb640 1 -- 172.21.15.28:0/2906653145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc0ab770 con 0x7fb51406c9d0 2023-12-16T21:58:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb511ffb640 1 -- 172.21.15.28:0/2906653145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc0abb00 con 0x7fb51406c9d0 2023-12-16T21:58:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- 172.21.15.28:0/2906653145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51406c9d0 msgr2=0x7fb514070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 --2- 172.21.15.28:0/2906653145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51406c9d0 0x7fb514070460 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0097c0 tx=0x7fb4fc01c410 comp rx=0 tx=0).stop 2023-12-16T21:58:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- 172.21.15.28:0/2906653145 shutdown_connections 2023-12-16T21:58:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 --2- 172.21.15.28:0/2906653145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb51406c9d0 0x7fb514070460 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- 172.21.15.28:0/2906653145 >> 172.21.15.28:0/2906653145 conn(0x7fb51406a530 msgr2=0x7fb51406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- 172.21.15.28:0/2906653145 shutdown_connections 2023-12-16T21:58:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- 172.21.15.28:0/2906653145 wait complete. 2023-12-16T21:58:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 Processor -- start 2023-12-16T21:58:17.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- start start 2023-12-16T21:58:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb51406c9d0 0x7fb51413acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb514072180 0x7fb514162800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb514144840 0x7fb514162d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb514166180 con 0x7fb514072180 2023-12-16T21:58:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb514143060 con 0x7fb514144840 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5192b9640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5141422a0 con 0x7fb51406c9d0 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb514072180 0x7fb514162800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb514072180 0x7fb514162800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60622/0 (socket says 172.21.15.28:60622) 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 -- 172.21.15.28:0/3491209246 learned_addr learned my addr 172.21.15.28:0/3491209246 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5137fe640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb514144840 0x7fb514162d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb512ffd640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb51406c9d0 0x7fb51413acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 -- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb51406c9d0 msgr2=0x7fb51413acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb51406c9d0 0x7fb51413acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 -- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb514144840 msgr2=0x7fb514162d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb514144840 0x7fb514162d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 -- 172.21.15.28:0/3491209246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc009420 con 0x7fb514072180 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5137fe640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb514144840 0x7fb514162d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb512ffd640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb51406c9d0 0x7fb51413acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.282+0000 7fb5127fc640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb514072180 0x7fb514162800 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0ad040 tx=0x7fb4fc01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb50b7fe640 1 -- 172.21.15.28:0/3491209246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc0b5810 con 0x7fb514072180 2023-12-16T21:58:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb50b7fe640 1 -- 172.21.15.28:0/3491209246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc0b5990 con 0x7fb514072180 2023-12-16T21:58:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb514142580 con 0x7fb514072180 2023-12-16T21:58:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb50b7fe640 1 -- 172.21.15.28:0/3491209246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc0b5d40 con 0x7fb514072180 2023-12-16T21:58:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb514142b40 con 0x7fb514072180 2023-12-16T21:58:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb514069440 con 0x7fb514072180 2023-12-16T21:58:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb50b7fe640 1 -- 172.21.15.28:0/3491209246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fb4fc0ca430 con 0x7fb514072180 2023-12-16T21:58:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb50b7fe640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb4f4066470 0x7fb4f4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb50b7fe640 1 -- 172.21.15.28:0/3491209246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4fc11ed10 con 0x7fb514072180 2023-12-16T21:58:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb512ffd640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb4f4066470 0x7fb4f4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.286+0000 7fb512ffd640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb4f4066470 0x7fb4f4068930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb504007e90 tx=0x7fb5040073a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.290+0000 7fb50b7fe640 1 -- 172.21.15.28:0/3491209246 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fb4fc0ec200 con 0x7fb514072180 2023-12-16T21:58:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.454+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb51415c530 con 0x7fb514072180 2023-12-16T21:58:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.454+0000 7fb50b7fe640 1 -- 172.21.15.28:0/3491209246 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb51415c530 con 0x7fb514072180 2023-12-16T21:58:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb4f4066470 msgr2=0x7fb4f4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb4f4066470 0x7fb4f4068930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb504007e90 tx=0x7fb5040073a0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb514072180 msgr2=0x7fb514162800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb514072180 0x7fb514162800 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0ad040 tx=0x7fb4fc01ac50 comp rx=0 tx=0).stop 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 shutdown_connections 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb4f4066470 0x7fb4f4068930 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb514144840 0x7fb514162d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb514072180 0x7fb514162800 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 --2- 172.21.15.28:0/3491209246 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb51406c9d0 0x7fb51413acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 >> 172.21.15.28:0/3491209246 conn(0x7fb51406a530 msgr2=0x7fb51413c050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 shutdown_connections 2023-12-16T21:58:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.458+0000 7fb5192b9640 1 -- 172.21.15.28:0/3491209246 wait complete. 2023-12-16T21:58:17.475 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2023-12-16T21:58:17.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:17.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:17.494 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:58:17.495 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2023-12-16T21:58:17.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:17.495 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:58:17.542 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:58:17.542 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:17.549 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:58:17.549 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:17.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.714+0000 7f2f23962640 1 Processor -- start 2023-12-16T21:58:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.714+0000 7f2f23962640 1 -- start start 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f23962640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2f1c15d240 0x7f2f1c0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f23962640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1c0adca0 con 0x7f2f1c06cda0 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f23962640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1c0adf80 con 0x7f2f1c06bc50 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f23962640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f1c0ae260 con 0x7f2f1c15d240 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f216fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2f1c15d240 0x7f2f1c0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f20efd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2f1c06cda0 0x7f2f1c06b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47440/0 (socket says 172.21.15.28:47440) 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f20efd640 1 -- 172.21.15.28:0/2138655814 learned_addr learned my addr 172.21.15.28:0/2138655814 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f216fe640 1 -- 172.21.15.28:0/2138655814 >> v1:172.21.15.122:6789/0 conn(0x7f2f1c06bc50 legacy=0x7f2f1c0b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f216fe640 1 -- 172.21.15.28:0/2138655814 >> v1:172.21.15.28:6789/0 conn(0x7f2f1c06cda0 legacy=0x7f2f1c06b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f13fff640 1 -- 172.21.15.28:0/2138655814 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013751741 0 0) 0x7f2f1c0adca0 con 0x7f2f1c06cda0 2023-12-16T21:58:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.718+0000 7f2f216fe640 1 -- 172.21.15.28:0/2138655814 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c0ae540 con 0x7f2f1c15d240 2023-12-16T21:58:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f216fe640 1 --2- 172.21.15.28:0/2138655814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c15d240 0x7f2f1c0aed40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2f180097c0 tx=0x7f2f1c0b3710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9480e2d6e19337d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f13fff640 1 -- 172.21.15.28:0/2138655814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180ad660 con 0x7f2f1c15d240 2023-12-16T21:58:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f13fff640 1 -- 172.21.15.28:0/2138655814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f180ad7c0 con 0x7f2f1c15d240 2023-12-16T21:58:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f13fff640 1 -- 172.21.15.28:0/2138655814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180adb50 con 0x7f2f1c15d240 2023-12-16T21:58:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f23962640 1 -- 172.21.15.28:0/2138655814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c15d240 msgr2=0x7f2f1c0aed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f23962640 1 --2- 172.21.15.28:0/2138655814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c15d240 0x7f2f1c0aed40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2f180097c0 tx=0x7f2f1c0b3710 comp rx=0 tx=0).stop 2023-12-16T21:58:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f23962640 1 -- 172.21.15.28:0/2138655814 shutdown_connections 2023-12-16T21:58:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f23962640 1 --2- 172.21.15.28:0/2138655814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c15d240 0x7f2f1c0aed40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f23962640 1 -- 172.21.15.28:0/2138655814 >> 172.21.15.28:0/2138655814 conn(0x7f2f1c070d40 msgr2=0x7f2f1c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f23962640 1 -- 172.21.15.28:0/2138655814 shutdown_connections 2023-12-16T21:58:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.722+0000 7f2f23962640 1 -- 172.21.15.28:0/2138655814 wait complete. 2023-12-16T21:58:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 Processor -- start 2023-12-16T21:58:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 -- start start 2023-12-16T21:58:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c06bc50 0x7f2f1c14ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c06cda0 0x7f2f1c1481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f1c15d240 0x7f2f1c148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c0afda0 con 0x7f2f1c06cda0 2023-12-16T21:58:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c15bdf0 con 0x7f2f1c06bc50 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c0af470 con 0x7f2f1c15d240 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c06cda0 0x7f2f1c1481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c06cda0 0x7f2f1c1481d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46418/0 (socket says 172.21.15.28:46418) 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 -- 172.21.15.28:0/2437254541 learned_addr learned my addr 172.21.15.28:0/2437254541 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f216fe640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c06bc50 0x7f2f1c14ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f21eff640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f1c15d240 0x7f2f1c148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 -- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f1c15d240 msgr2=0x7f2f1c148710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f1c15d240 0x7f2f1c148710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 -- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c06bc50 msgr2=0x7f2f1c14ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c06bc50 0x7f2f1c14ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 -- 172.21.15.28:0/2437254541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f18009420 con 0x7f2f1c06cda0 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f216fe640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c06bc50 0x7f2f1c14ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f20efd640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c06cda0 0x7f2f1c1481d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00c4a0 tx=0x7f2f0c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f11ffb640 1 -- 172.21.15.28:0/2437254541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c0074e0 con 0x7f2f1c06cda0 2023-12-16T21:58:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f11ffb640 1 -- 172.21.15.28:0/2437254541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c007640 con 0x7f2f1c06cda0 2023-12-16T21:58:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1c0af750 con 0x7f2f1c06cda0 2023-12-16T21:58:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f11ffb640 1 -- 172.21.15.28:0/2437254541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c004a30 con 0x7f2f1c06cda0 2023-12-16T21:58:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1c151540 con 0x7f2f1c06cda0 2023-12-16T21:58:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1c0aff10 con 0x7f2f1c06cda0 2023-12-16T21:58:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f11ffb640 1 -- 172.21.15.28:0/2437254541 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f2f0c004b90 con 0x7f2f1c06cda0 2023-12-16T21:58:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f11ffb640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f00066470 0x7f2f00068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.726+0000 7f2f11ffb640 1 -- 172.21.15.28:0/2437254541 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f0c083080 con 0x7f2f1c06cda0 2023-12-16T21:58:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.730+0000 7f2f216fe640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f00066470 0x7f2f00068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.730+0000 7f2f216fe640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f00066470 0x7f2f00068930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f180af000 tx=0x7f2f18004740 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.730+0000 7f2f11ffb640 1 -- 172.21.15.28:0/2437254541 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f2f0c050590 con 0x7f2f1c06cda0 2023-12-16T21:58:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.890+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f1c0674d0 con 0x7f2f1c06cda0 2023-12-16T21:58:17.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.890+0000 7f2f11ffb640 1 -- 172.21.15.28:0/2437254541 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f2f0c018090 con 0x7f2f1c06cda0 2023-12-16T21:58:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:17.895 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f00066470 msgr2=0x7f2f00068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f00066470 0x7f2f00068930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f180af000 tx=0x7f2f18004740 comp rx=0 tx=0).stop 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c06cda0 msgr2=0x7f2f1c1481d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c06cda0 0x7f2f1c1481d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00c4a0 tx=0x7f2f0c00c960 comp rx=0 tx=0).stop 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 shutdown_connections 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f00066470 0x7f2f00068930 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f1c15d240 0x7f2f1c148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f1c06cda0 0x7f2f1c1481d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 --2- 172.21.15.28:0/2437254541 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c06bc50 0x7f2f1c14ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 >> 172.21.15.28:0/2437254541 conn(0x7f2f1c070d40 msgr2=0x7f2f1c15dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 shutdown_connections 2023-12-16T21:58:17.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:17.894+0000 7f2f23962640 1 -- 172.21.15.28:0/2437254541 wait complete. 2023-12-16T21:58:17.914 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:17.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:18.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.038+0000 7f487a4f2640 1 Processor -- start 2023-12-16T21:58:18.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.038+0000 7f487a4f2640 1 -- start start 2023-12-16T21:58:18.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f487a4f2640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48740b42d0 0x7f48740b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f487a4f2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48740b4c60 con 0x7f48740cba10 2023-12-16T21:58:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f487a4f2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487415b670 con 0x7f4874158ca0 2023-12-16T21:58:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f487a4f2640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487415c510 con 0x7f48740b42d0 2023-12-16T21:58:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f48740cba10 0x7f487415be00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47446/0 (socket says 172.21.15.28:47446) 2023-12-16T21:58:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 -- 172.21.15.28:0/1610655011 learned_addr learned my addr 172.21.15.28:0/1610655011 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4878a8f640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48740b42d0 0x7f48740b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155255768 0 0) 0x7f4868001120 con 0x7f48740cba10 2023-12-16T21:58:18.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485c003530 con 0x7f48740cba10 2023-12-16T21:58:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2279138286 0 0) 0x7f485c003530 con 0x7f48740cba10 2023-12-16T21:58:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48740b42d0 msgr2=0x7f48740b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:18.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48740b42d0 0x7f48740b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 >> v1:172.21.15.122:6789/0 conn(0x7f4874158ca0 legacy=0x7f487415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487415f410 con 0x7f48740cba10 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f4868001810 con 0x7f48740cba10 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 >> v1:172.21.15.28:6789/0 conn(0x7f48740cba10 legacy=0x7f487415be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485c0039b0 0x7f485c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485c004360 0x7f485c004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48740b42d0 0x7f485c005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487415c510 con 0x7f485c004360 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485c00b9c0 con 0x7f485c0039b0 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485c00bb80 con 0x7f48740b42d0 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485c004360 0x7f485c004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4873fff640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485c0039b0 0x7f485c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4878a8f640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48740b42d0 0x7f485c005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 -- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48740b42d0 msgr2=0x7f485c005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:18.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48740b42d0 0x7f485c005090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 -- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485c0039b0 msgr2=0x7f485c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:18.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485c0039b0 0x7f485c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 -- 172.21.15.28:0/1610655011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c003530 con 0x7f485c004360 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f48737fe640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485c004360 0x7f485c004830 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f486800c190 tx=0x7f486801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca784791936784bb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48680a5b00 con 0x7f485c004360 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48680a5c80 con 0x7f485c004360 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.042+0000 7f4872ffd640 1 -- 172.21.15.28:0/1610655011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48680058e0 con 0x7f485c004360 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- 172.21.15.28:0/1610655011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485c004360 msgr2=0x7f485c004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 --2- 172.21.15.28:0/1610655011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f485c004360 0x7f485c004830 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f486800c190 tx=0x7f486801cc00 comp rx=0 tx=0).stop 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 -- 172.21.15.28:0/1610655011 reap_dead start 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- 172.21.15.28:0/1610655011 shutdown_connections 2023-12-16T21:58:18.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- 172.21.15.28:0/1610655011 >> 172.21.15.28:0/1610655011 conn(0x7f48740b0cc0 msgr2=0x7f48740b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:18.049 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- 172.21.15.28:0/1610655011 shutdown_connections 2023-12-16T21:58:18.049 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- 172.21.15.28:0/1610655011 wait complete. 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 Processor -- start 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- start start 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f485c004360 0x7f48741468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f487414f130 0x7f4874153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4874153c40 0x7f48740b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48741540c0 con 0x7f4874153c40 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4874146e10 con 0x7f487414f130 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48740b20c0 con 0x7f485c004360 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4874153c40 0x7f48740b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4874153c40 0x7f48740b2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46438/0 (socket says 172.21.15.28:46438) 2023-12-16T21:58:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 -- 172.21.15.28:0/2143334400 learned_addr learned my addr 172.21.15.28:0/2143334400 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4873fff640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f485c004360 0x7f48741468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f48737fe640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f487414f130 0x7f4874153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 -- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f485c004360 msgr2=0x7f48741468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f485c004360 0x7f48741468d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 -- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f487414f130 msgr2=0x7f4874153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f487414f130 0x7f4874153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 -- 172.21.15.28:0/2143334400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48740b2680 con 0x7f4874153c40 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4878a8f640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4874153c40 0x7f48740b2b10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f486c009fd0 tx=0x7f486c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4870ff9640 1 -- 172.21.15.28:0/2143334400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c017400 con 0x7f4874153c40 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4870ff9640 1 -- 172.21.15.28:0/2143334400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c011040 con 0x7f4874153c40 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4874157420 con 0x7f4874153c40 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f4870ff9640 1 -- 172.21.15.28:0/2143334400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c0177c0 con 0x7f4874153c40 2023-12-16T21:58:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.046+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48741579e0 con 0x7f4874153c40 2023-12-16T21:58:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.050+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48740b2810 con 0x7f4874153c40 2023-12-16T21:58:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.050+0000 7f4870ff9640 1 -- 172.21.15.28:0/2143334400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f486c0203b0 con 0x7f4874153c40 2023-12-16T21:58:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.050+0000 7f4870ff9640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4860066470 0x7f4860068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.050+0000 7f4870ff9640 1 -- 172.21.15.28:0/2143334400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f486c084200 con 0x7f4874153c40 2023-12-16T21:58:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.050+0000 7f4873fff640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4860066470 0x7f4860068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.050+0000 7f4873fff640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4860066470 0x7f4860068930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f48740cb880 tx=0x7f486400a910 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.054+0000 7f4870ff9640 1 -- 172.21.15.28:0/2143334400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f486c051710 con 0x7f4874153c40 2023-12-16T21:58:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.202+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48740693d0 con 0x7f4874153c40 2023-12-16T21:58:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.202+0000 7f4870ff9640 1 -- 172.21.15.28:0/2143334400 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f486c054d30 con 0x7f4874153c40 2023-12-16T21:58:18.206 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4860066470 msgr2=0x7f4860068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4860066470 0x7f4860068930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f48740cb880 tx=0x7f486400a910 comp rx=0 tx=0).stop 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4874153c40 msgr2=0x7f48740b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4874153c40 0x7f48740b2b10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f486c009fd0 tx=0x7f486c00f4a0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 shutdown_connections 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4860066470 0x7f4860068930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4874153c40 0x7f48740b2b10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f487414f130 0x7f4874153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 --2- 172.21.15.28:0/2143334400 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f485c004360 0x7f48741468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 >> 172.21.15.28:0/2143334400 conn(0x7f48740b0cc0 msgr2=0x7f487415a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 shutdown_connections 2023-12-16T21:58:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.206+0000 7f487a4f2640 1 -- 172.21.15.28:0/2143334400 wait complete. 2023-12-16T21:58:18.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:18.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710e5b7640 1 Processor -- start 2023-12-16T21:58:18.342 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710e5b7640 1 -- start start 2023-12-16T21:58:18.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710e5b7640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7108158ca0 0x7f710815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710e5b7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71080b42d0 0x7f71080b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710e5b7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71080b4c40 con 0x7f71080cba10 2023-12-16T21:58:18.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710e5b7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f710815b670 con 0x7f71080b42d0 2023-12-16T21:58:18.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710e5b7640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710815c510 con 0x7f7108158ca0 2023-12-16T21:58:18.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710cb54640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71080b42d0 0x7f71080b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710cb54640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71080b42d0 0x7f71080b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46446/0 (socket says 172.21.15.28:46446) 2023-12-16T21:58:18.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f710cb54640 1 -- 172.21.15.28:0/1221868839 learned_addr learned my addr 172.21.15.28:0/1221868839 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:18.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.338+0000 7f7107fff640 1 --2- 172.21.15.28:0/1221868839 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7108158ca0 0x7f710815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710cb54640 1 -- 172.21.15.28:0/1221868839 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7108158ca0 msgr2=0x7f710815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:18.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710cb54640 1 --2- 172.21.15.28:0/1221868839 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7108158ca0 0x7f710815eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710cb54640 1 -- 172.21.15.28:0/1221868839 >> v1:172.21.15.122:6789/0 conn(0x7f71080cba10 legacy=0x7f710815be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:18.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710cb54640 1 -- 172.21.15.28:0/1221868839 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710815f410 con 0x7f71080b42d0 2023-12-16T21:58:18.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710cb54640 1 --2- 172.21.15.28:0/1221868839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71080b42d0 0x7f71080b4700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f71080725c0 tx=0x7f710001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351c1a943bba6634 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f7106ffd640 1 -- 172.21.15.28:0/1221868839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000a5800 con 0x7f71080b42d0 2023-12-16T21:58:18.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f7106ffd640 1 -- 172.21.15.28:0/1221868839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71000a5960 con 0x7f71080b42d0 2023-12-16T21:58:18.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f7106ffd640 1 -- 172.21.15.28:0/1221868839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000a5cf0 con 0x7f71080b42d0 2023-12-16T21:58:18.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710e5b7640 1 -- 172.21.15.28:0/1221868839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71080b42d0 msgr2=0x7f71080b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710e5b7640 1 --2- 172.21.15.28:0/1221868839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71080b42d0 0x7f71080b4700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f71080725c0 tx=0x7f710001c980 comp rx=0 tx=0).stop 2023-12-16T21:58:18.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710e5b7640 1 -- 172.21.15.28:0/1221868839 shutdown_connections 2023-12-16T21:58:18.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710e5b7640 1 --2- 172.21.15.28:0/1221868839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71080b42d0 0x7f71080b4700 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710e5b7640 1 --2- 172.21.15.28:0/1221868839 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7108158ca0 0x7f710815eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710e5b7640 1 -- 172.21.15.28:0/1221868839 >> 172.21.15.28:0/1221868839 conn(0x7f71080b0cc0 msgr2=0x7f71080b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:18.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710e5b7640 1 -- 172.21.15.28:0/1221868839 shutdown_connections 2023-12-16T21:58:18.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.342+0000 7f710e5b7640 1 -- 172.21.15.28:0/1221868839 wait complete. 2023-12-16T21:58:18.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 Processor -- start 2023-12-16T21:58:18.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 -- start start 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71080b42d0 0x7f710813b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71080cba10 0x7f71080cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7108158ca0 0x7f71080cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7108151390 con 0x7f7108158ca0 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7108151670 con 0x7f71080cba10 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710815c6e0 con 0x7f71080b42d0 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7108158ca0 0x7f71080cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7108158ca0 0x7f71080cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46454/0 (socket says 172.21.15.28:46454) 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 -- 172.21.15.28:0/465172793 learned_addr learned my addr 172.21.15.28:0/465172793 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:18.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f71077fe640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71080cba10 0x7f71080cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7107fff640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71080b42d0 0x7f710813b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 -- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71080b42d0 msgr2=0x7f710813b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71080b42d0 0x7f710813b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 -- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71080cba10 msgr2=0x7f71080cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71080cba10 0x7f71080cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 -- 172.21.15.28:0/465172793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7100009b30 con 0x7f7108158ca0 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7107fff640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71080b42d0 0x7f710813b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f71077fe640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71080cba10 0x7f71080cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710cb54640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7108158ca0 0x7f71080cc3c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f71000af040 tx=0x7f7100019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7104ff9640 1 -- 172.21.15.28:0/465172793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000b8670 con 0x7f7108158ca0 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7104ff9640 1 -- 172.21.15.28:0/465172793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71000b87d0 con 0x7f7108158ca0 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710815c9c0 con 0x7f7108158ca0 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7104ff9640 1 -- 172.21.15.28:0/465172793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000b8b80 con 0x7f7108158ca0 2023-12-16T21:58:18.350 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710813bc60 con 0x7f7108158ca0 2023-12-16T21:58:18.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71080693d0 con 0x7f7108158ca0 2023-12-16T21:58:18.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7104ff9640 1 -- 172.21.15.28:0/465172793 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f71000b8ce0 con 0x7f7108158ca0 2023-12-16T21:58:18.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7104ff9640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f70e8066470 0x7f70e8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7104ff9640 1 -- 172.21.15.28:0/465172793 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f710011f450 con 0x7f7108158ca0 2023-12-16T21:58:18.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.346+0000 7f7107fff640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f70e8066470 0x7f70e8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.350+0000 7f7107fff640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f70e8066470 0x7f70e8068930 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009840 tx=0x7f70f8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.350+0000 7f7104ff9640 1 -- 172.21.15.28:0/465172793 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f71000ec9f0 con 0x7f7108158ca0 2023-12-16T21:58:18.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.506+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f71080693d0 con 0x7f7108158ca0 2023-12-16T21:58:18.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.506+0000 7f7104ff9640 1 -- 172.21.15.28:0/465172793 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f71000f0010 con 0x7f7108158ca0 2023-12-16T21:58:18.511 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:18.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f70e8066470 msgr2=0x7f70e8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f70e8066470 0x7f70e8068930 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009840 tx=0x7f70f8008040 comp rx=0 tx=0).stop 2023-12-16T21:58:18.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7108158ca0 msgr2=0x7f71080cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7108158ca0 0x7f71080cc3c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f71000af040 tx=0x7f7100019720 comp rx=0 tx=0).stop 2023-12-16T21:58:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 shutdown_connections 2023-12-16T21:58:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f70e8066470 0x7f70e8068930 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7108158ca0 0x7f71080cc3c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71080cba10 0x7f71080cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 --2- 172.21.15.28:0/465172793 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71080b42d0 0x7f710813b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 >> 172.21.15.28:0/465172793 conn(0x7f71080b0cc0 msgr2=0x7f71081595a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 shutdown_connections 2023-12-16T21:58:18.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.510+0000 7f710e5b7640 1 -- 172.21.15.28:0/465172793 wait complete. 2023-12-16T21:58:18.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f407872f640 1 Processor -- start 2023-12-16T21:58:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f407872f640 1 -- start start 2023-12-16T21:58:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f407872f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40700cba10 0x7f407015be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f407872f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f40700b42d0 0x7f40700b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f407872f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40700b4c60 con 0x7f4070158ca0 2023-12-16T21:58:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f407872f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407015b670 con 0x7f40700cba10 2023-12-16T21:58:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f407872f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407015f410 con 0x7f40700b42d0 2023-12-16T21:58:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f40764cb640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4070158ca0 0x7f407015eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47456/0 (socket says 172.21.15.28:47456) 2023-12-16T21:58:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f40764cb640 1 -- 172.21.15.28:0/3753659590 learned_addr learned my addr 172.21.15.28:0/3753659590 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f4075cca640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40700cba10 0x7f407015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f40754c9640 1 -- 172.21.15.28:0/3753659590 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019426447 0 0) 0x7f406c0014a0 con 0x7f4070158ca0 2023-12-16T21:58:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f4076ccc640 1 --2- 172.21.15.28:0/3753659590 >> v2:172.21.15.122:3300/0 conn(0x7f40700b42d0 0x7f40700b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f40754c9640 1 -- 172.21.15.28:0/3753659590 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c003530 con 0x7f4070158ca0 2023-12-16T21:58:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f4075cca640 1 -- 172.21.15.28:0/3753659590 >> v2:172.21.15.122:3300/0 conn(0x7f40700b42d0 msgr2=0x7f40700b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f4075cca640 1 --2- 172.21.15.28:0/3753659590 >> v2:172.21.15.122:3300/0 conn(0x7f40700b42d0 0x7f40700b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f4075cca640 1 -- 172.21.15.28:0/3753659590 >> v1:172.21.15.28:6789/0 conn(0x7f4070158ca0 legacy=0x7f407015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.658+0000 7f4075cca640 1 -- 172.21.15.28:0/3753659590 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407015f6f0 con 0x7f40700cba10 2023-12-16T21:58:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f4075cca640 1 --2- 172.21.15.28:0/3753659590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 0x7f407015be00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40700cb970 tx=0x7f406001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df5d56475427cef server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f40754c9640 1 -- 172.21.15.28:0/3753659590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40600ab600 con 0x7f40700cba10 2023-12-16T21:58:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f40754c9640 1 -- 172.21.15.28:0/3753659590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40600a7050 con 0x7f40700cba10 2023-12-16T21:58:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f40754c9640 1 -- 172.21.15.28:0/3753659590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40600ab980 con 0x7f40700cba10 2023-12-16T21:58:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- 172.21.15.28:0/3753659590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 msgr2=0x7f407015be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 --2- 172.21.15.28:0/3753659590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 0x7f407015be00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40700cb970 tx=0x7f406001c460 comp rx=0 tx=0).stop 2023-12-16T21:58:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- 172.21.15.28:0/3753659590 shutdown_connections 2023-12-16T21:58:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 --2- 172.21.15.28:0/3753659590 >> v2:172.21.15.122:3300/0 conn(0x7f40700b42d0 0x7f40700b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 --2- 172.21.15.28:0/3753659590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 0x7f407015be00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- 172.21.15.28:0/3753659590 >> 172.21.15.28:0/3753659590 conn(0x7f40700b0cc0 msgr2=0x7f40700b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- 172.21.15.28:0/3753659590 shutdown_connections 2023-12-16T21:58:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- 172.21.15.28:0/3753659590 wait complete. 2023-12-16T21:58:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 Processor -- start 2023-12-16T21:58:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- start start 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40700b42d0 0x7f4070145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 0x7f4070165550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4070158ca0 0x7f40700cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40700b2d70 con 0x7f40700cba10 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40701461b0 con 0x7f40700b42d0 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f407872f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40700b2160 con 0x7f4070158ca0 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f40764cb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40700b42d0 0x7f4070145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f4075cca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 0x7f4070165550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f40764cb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40700b42d0 0x7f4070145c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:37378/0 (socket says 172.21.15.28:37378) 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.662+0000 7f4076ccc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4070158ca0 0x7f40700cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f40764cb640 1 -- 172.21.15.28:0/1241989642 learned_addr learned my addr 172.21.15.28:0/1241989642 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4075cca640 1 -- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4070158ca0 msgr2=0x7f40700cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4075cca640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4070158ca0 0x7f40700cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4075cca640 1 -- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40700b42d0 msgr2=0x7f4070145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4075cca640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40700b42d0 0x7f4070145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4075cca640 1 -- 172.21.15.28:0/1241989642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060009420 con 0x7f40700cba10 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4075cca640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 0x7f4070165550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f40600ab000 tx=0x7f4060019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4066ffd640 1 -- 172.21.15.28:0/1241989642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40600b9070 con 0x7f40700cba10 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4066ffd640 1 -- 172.21.15.28:0/1241989642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4060006e20 con 0x7f40700cba10 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40700b2440 con 0x7f40700cba10 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4066ffd640 1 -- 172.21.15.28:0/1241989642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40600b4880 con 0x7f40700cba10 2023-12-16T21:58:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40700b2a00 con 0x7f40700cba10 2023-12-16T21:58:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40700b50f0 con 0x7f40700cba10 2023-12-16T21:58:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4066ffd640 1 -- 172.21.15.28:0/1241989642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f40600b2040 con 0x7f40700cba10 2023-12-16T21:58:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4066ffd640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f40540664c0 0x7f4054068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f4066ffd640 1 -- 172.21.15.28:0/1241989642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f406011d5d0 con 0x7f40700cba10 2023-12-16T21:58:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f40764cb640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f40540664c0 0x7f4054068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.666+0000 7f40764cb640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f40540664c0 0x7f4054068980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f406c001700 tx=0x7f406c001ad0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.670+0000 7f4066ffd640 1 -- 172.21.15.28:0/1241989642 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f40600eac70 con 0x7f40700cba10 2023-12-16T21:58:18.831 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.826+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f40700693d0 con 0x7f40700cba10 2023-12-16T21:58:18.831 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f4066ffd640 1 -- 172.21.15.28:0/1241989642 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f40600ee290 con 0x7f40700cba10 2023-12-16T21:58:18.832 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f40540664c0 msgr2=0x7f4054068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f40540664c0 0x7f4054068980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f406c001700 tx=0x7f406c001ad0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 msgr2=0x7f4070165550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 0x7f4070165550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f40600ab000 tx=0x7f4060019200 comp rx=0 tx=0).stop 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 shutdown_connections 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f40540664c0 0x7f4054068980 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4070158ca0 0x7f40700cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40700cba10 0x7f4070165550 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 --2- 172.21.15.28:0/1241989642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f40700b42d0 0x7f4070145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 >> 172.21.15.28:0/1241989642 conn(0x7f40700b0cc0 msgr2=0x7f4070159080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:18.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 shutdown_connections 2023-12-16T21:58:18.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.830+0000 7f407872f640 1 -- 172.21.15.28:0/1241989642 wait complete. 2023-12-16T21:58:18.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:58:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d096a4640 1 Processor -- start 2023-12-16T21:58:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d096a4640 1 -- start start 2023-12-16T21:58:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d096a4640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d04144840 0x7f5d0406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d096a4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5d04072180 0x7f5d0406c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d096a4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d040710e0 con 0x7f5d04070460 2023-12-16T21:58:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d096a4640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0415f010 con 0x7f5d04144840 2023-12-16T21:58:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d096a4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d040713c0 con 0x7f5d04072180 2023-12-16T21:58:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d02ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d04144840 0x7f5d0406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d02ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d04144840 0x7f5d0406c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46498/0 (socket says 172.21.15.28:46498) 2023-12-16T21:58:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d02ffd640 1 -- 172.21.15.28:0/1603538472 learned_addr learned my addr 172.21.15.28:0/1603538472 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d027fc640 1 --2- 172.21.15.28:0/1603538472 >> v2:172.21.15.122:3300/0 conn(0x7f5d04072180 0x7f5d0406c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d02ffd640 1 -- 172.21.15.28:0/1603538472 >> v2:172.21.15.122:3300/0 conn(0x7f5d04072180 msgr2=0x7f5d0406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d02ffd640 1 --2- 172.21.15.28:0/1603538472 >> v2:172.21.15.122:3300/0 conn(0x7f5d04072180 0x7f5d0406c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d02ffd640 1 -- 172.21.15.28:0/1603538472 >> v1:172.21.15.122:6789/0 conn(0x7f5d04070460 legacy=0x7f5d04070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.966+0000 7f5d02ffd640 1 -- 172.21.15.28:0/1603538472 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d040716a0 con 0x7f5d04144840 2023-12-16T21:58:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d02ffd640 1 --2- 172.21.15.28:0/1603538472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d0406c2c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5d0406d140 tx=0x7f5cf401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d71232767bd085 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d01ffb640 1 -- 172.21.15.28:0/1603538472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf40aa430 con 0x7f5d04144840 2023-12-16T21:58:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d01ffb640 1 -- 172.21.15.28:0/1603538472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf40aa590 con 0x7f5d04144840 2023-12-16T21:58:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d01ffb640 1 -- 172.21.15.28:0/1603538472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf40aa920 con 0x7f5d04144840 2023-12-16T21:58:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d096a4640 1 -- 172.21.15.28:0/1603538472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 msgr2=0x7f5d0406c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1603538472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d0406c2c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5d0406d140 tx=0x7f5cf401c4d0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d096a4640 1 -- 172.21.15.28:0/1603538472 shutdown_connections 2023-12-16T21:58:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1603538472 >> v2:172.21.15.122:3300/0 conn(0x7f5d04072180 0x7f5d0406c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1603538472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d0406c2c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d096a4640 1 -- 172.21.15.28:0/1603538472 >> 172.21.15.28:0/1603538472 conn(0x7f5d0406a530 msgr2=0x7f5d0406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d096a4640 1 -- 172.21.15.28:0/1603538472 shutdown_connections 2023-12-16T21:58:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.970+0000 7f5d096a4640 1 -- 172.21.15.28:0/1603538472 wait complete. 2023-12-16T21:58:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 Processor -- start 2023-12-16T21:58:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 -- start start 2023-12-16T21:58:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d04070460 0x7f5d04164de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5d04072180 0x7f5d04158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d04141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d04138c80 con 0x7f5d04144840 2023-12-16T21:58:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d04138630 con 0x7f5d04070460 2023-12-16T21:58:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d04141590 con 0x7f5d04072180 2023-12-16T21:58:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d04141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d04141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46508/0 (socket says 172.21.15.28:46508) 2023-12-16T21:58:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 -- 172.21.15.28:0/1542184913 learned_addr learned my addr 172.21.15.28:0/1542184913 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d02ffd640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d04070460 0x7f5d04164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d027fc640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5d04072180 0x7f5d04158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 -- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5d04072180 msgr2=0x7f5d04158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5d04072180 0x7f5d04158430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 -- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d04070460 msgr2=0x7f5d04164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d04070460 0x7f5d04164de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 -- 172.21.15.28:0/1542184913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf40094d0 con 0x7f5d04144840 2023-12-16T21:58:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d037fe640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d04141d70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c770 tx=0x7f5cfc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5ceb7fe640 1 -- 172.21.15.28:0/1542184913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc003b80 con 0x7f5d04144840 2023-12-16T21:58:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5ceb7fe640 1 -- 172.21.15.28:0/1542184913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc003ce0 con 0x7f5d04144840 2023-12-16T21:58:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0415e050 con 0x7f5d04144840 2023-12-16T21:58:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5ceb7fe640 1 -- 172.21.15.28:0/1542184913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc00aa90 con 0x7f5d04144840 2023-12-16T21:58:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d04165320 con 0x7f5d04144840 2023-12-16T21:58:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d04069440 con 0x7f5d04144840 2023-12-16T21:58:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5ceb7fe640 1 -- 172.21.15.28:0/1542184913 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f5cfc00abf0 con 0x7f5d04144840 2023-12-16T21:58:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.974+0000 7f5ceb7fe640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5cf00664a0 0x7f5cf0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.978+0000 7f5ceb7fe640 1 -- 172.21.15.28:0/1542184913 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cfc083c00 con 0x7f5d04144840 2023-12-16T21:58:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.978+0000 7f5d02ffd640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5cf00664a0 0x7f5cf0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.978+0000 7f5d02ffd640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5cf00664a0 0x7f5cf0068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40ae000 tx=0x7f5cf4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:18.982+0000 7f5ceb7fe640 1 -- 172.21.15.28:0/1542184913 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f5cfc08a080 con 0x7f5d04144840 2023-12-16T21:58:19.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.146+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5d0406d170 con 0x7f5d04144840 2023-12-16T21:58:19.152 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5ceb7fe640 1 -- 172.21.15.28:0/1542184913 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5cfc051110 con 0x7f5d04144840 2023-12-16T21:58:19.152 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2023-12-16T21:58:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5cf00664a0 msgr2=0x7f5cf0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5cf00664a0 0x7f5cf0068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40ae000 tx=0x7f5cf4008040 comp rx=0 tx=0).stop 2023-12-16T21:58:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 msgr2=0x7f5d04141d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d04141d70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc00c770 tx=0x7f5cfc00cc30 comp rx=0 tx=0).stop 2023-12-16T21:58:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 shutdown_connections 2023-12-16T21:58:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5cf00664a0 0x7f5cf0068960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d04144840 0x7f5d04141d70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5d04072180 0x7f5d04158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 --2- 172.21.15.28:0/1542184913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5d04070460 0x7f5d04164de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 >> 172.21.15.28:0/1542184913 conn(0x7f5d0406a530 msgr2=0x7f5d0406c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:19.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 shutdown_connections 2023-12-16T21:58:19.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.150+0000 7f5d096a4640 1 -- 172.21.15.28:0/1542184913 wait complete. 2023-12-16T21:58:19.170 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:58:19.170 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:19.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb936708640 1 Processor -- start 2023-12-16T21:58:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb936708640 1 -- start start 2023-12-16T21:58:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb936708640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300d2b10 0x7fb9300b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb936708640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb9300b51a0 0x7fb9300b5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb936708640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9300d0d70 con 0x7fb930160870 2023-12-16T21:58:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb936708640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb930160c50 con 0x7fb9300b51a0 2023-12-16T21:58:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb936708640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300b5b50 con 0x7fb9300d2b10 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb934ca5640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb9300b51a0 0x7fb9300b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb92ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb930160870 0x7fb9300cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47458/0 (socket says 172.21.15.28:47458) 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb92ffff640 1 -- 172.21.15.28:0/3784833651 learned_addr learned my addr 172.21.15.28:0/3784833651 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb92f7fe640 1 --2- 172.21.15.28:0/3784833651 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300d2b10 0x7fb9300b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb92effd640 1 -- 172.21.15.28:0/3784833651 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550827184 0 0) 0x7fb9300d0d70 con 0x7fb930160870 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb92effd640 1 -- 172.21.15.28:0/3784833651 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb91c003530 con 0x7fb930160870 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb934ca5640 1 -- 172.21.15.28:0/3784833651 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300d2b10 msgr2=0x7fb9300b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb934ca5640 1 --2- 172.21.15.28:0/3784833651 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300d2b10 0x7fb9300b6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb934ca5640 1 -- 172.21.15.28:0/3784833651 >> v1:172.21.15.28:6789/0 conn(0x7fb930160870 legacy=0x7fb9300cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb934ca5640 1 -- 172.21.15.28:0/3784833651 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300d2f50 con 0x7fb9300b51a0 2023-12-16T21:58:19.299 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.294+0000 7fb934ca5640 1 --2- 172.21.15.28:0/3784833651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300b51a0 0x7fb9300b5610 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb928009d90 tx=0x7fb92801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0cc42d841658da server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.299 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92effd640 1 -- 172.21.15.28:0/3784833651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9280a5900 con 0x7fb9300b51a0 2023-12-16T21:58:19.299 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92effd640 1 -- 172.21.15.28:0/3784833651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9280a5a60 con 0x7fb9300b51a0 2023-12-16T21:58:19.299 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92effd640 1 -- 172.21.15.28:0/3784833651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9280a5df0 con 0x7fb9300b51a0 2023-12-16T21:58:19.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- 172.21.15.28:0/3784833651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300b51a0 msgr2=0x7fb9300b5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 --2- 172.21.15.28:0/3784833651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300b51a0 0x7fb9300b5610 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb928009d90 tx=0x7fb92801c950 comp rx=0 tx=0).stop 2023-12-16T21:58:19.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- 172.21.15.28:0/3784833651 shutdown_connections 2023-12-16T21:58:19.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 --2- 172.21.15.28:0/3784833651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300b51a0 0x7fb9300b5610 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 --2- 172.21.15.28:0/3784833651 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300d2b10 0x7fb9300b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- 172.21.15.28:0/3784833651 >> 172.21.15.28:0/3784833651 conn(0x7fb930072060 msgr2=0x7fb930072470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:19.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- 172.21.15.28:0/3784833651 shutdown_connections 2023-12-16T21:58:19.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- 172.21.15.28:0/3784833651 wait complete. 2023-12-16T21:58:19.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 Processor -- start 2023-12-16T21:58:19.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- start start 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300b51a0 0x7fb930158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300d2b10 0x7fb9300c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb930160870 0x7fb9300c5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9301588c0 con 0x7fb9300d2b10 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300c5f40 con 0x7fb930160870 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb936708640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93015ac50 con 0x7fb9300b51a0 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300d2b10 0x7fb9300c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300d2b10 0x7fb9300c50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46526/0 (socket says 172.21.15.28:46526) 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 -- 172.21.15.28:0/4111738362 learned_addr learned my addr 172.21.15.28:0/4111738362 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92ffff640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300b51a0 0x7fb930158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb934ca5640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb930160870 0x7fb9300c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 -- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300b51a0 msgr2=0x7fb930158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300b51a0 0x7fb930158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 -- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb930160870 msgr2=0x7fb9300c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb930160870 0x7fb9300c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 -- 172.21.15.28:0/4111738362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb928009a70 con 0x7fb9300d2b10 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92f7fe640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300d2b10 0x7fb9300c50e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb91800d830 tx=0x7fb91800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.298+0000 7fb92cff9640 1 -- 172.21.15.28:0/4111738362 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800b660 con 0x7fb9300d2b10 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb92cff9640 1 -- 172.21.15.28:0/4111738362 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91800b7c0 con 0x7fb9300d2b10 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93015aec0 con 0x7fb9300d2b10 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb92cff9640 1 -- 172.21.15.28:0/4111738362 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800bb30 con 0x7fb9300d2b10 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93015b480 con 0x7fb9300d2b10 2023-12-16T21:58:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb930069440 con 0x7fb9300d2b10 2023-12-16T21:58:19.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb92cff9640 1 -- 172.21.15.28:0/4111738362 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fb91801e460 con 0x7fb9300d2b10 2023-12-16T21:58:19.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb92cff9640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb910066470 0x7fb910068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb92cff9640 1 -- 172.21.15.28:0/4111738362 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb918084500 con 0x7fb9300d2b10 2023-12-16T21:58:19.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb92ffff640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb910066470 0x7fb910068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.307 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.302+0000 7fb92ffff640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb910066470 0x7fb910068930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb9200011f0 tx=0x7fb920007d40 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.306+0000 7fb92cff9640 1 -- 172.21.15.28:0/4111738362 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fb918051990 con 0x7fb9300d2b10 2023-12-16T21:58:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.462+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb93015b900 con 0x7fb9300d2b10 2023-12-16T21:58:19.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.466+0000 7fb92cff9640 1 -- 172.21.15.28:0/4111738362 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fb918054fb0 con 0x7fb9300d2b10 2023-12-16T21:58:19.470 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:19.470 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb910066470 msgr2=0x7fb910068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb910066470 0x7fb910068930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb9200011f0 tx=0x7fb920007d40 comp rx=0 tx=0).stop 2023-12-16T21:58:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300d2b10 msgr2=0x7fb9300c50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300d2b10 0x7fb9300c50e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb91800d830 tx=0x7fb91800dcf0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 shutdown_connections 2023-12-16T21:58:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb910066470 0x7fb910068930 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb930160870 0x7fb9300c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300d2b10 0x7fb9300c50e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 --2- 172.21.15.28:0/4111738362 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb9300b51a0 0x7fb930158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 >> 172.21.15.28:0/4111738362 conn(0x7fb930072060 msgr2=0x7fb9300b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 shutdown_connections 2023-12-16T21:58:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.470+0000 7fb936708640 1 -- 172.21.15.28:0/4111738362 wait complete. 2023-12-16T21:58:19.489 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:19.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765c27a640 1 Processor -- start 2023-12-16T21:58:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765c27a640 1 -- start start 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765c27a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 0x7f765415be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765c27a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76540b42d0 0x7f76540b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765c27a640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76540b4c40 con 0x7f7654158ca0 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765c27a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765415b670 con 0x7f76540b42d0 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765c27a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765415f410 con 0x7f76540cba10 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765a817640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76540b42d0 0x7f76540b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765a016640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7654158ca0 0x7f765415eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47470/0 (socket says 172.21.15.28:47470) 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765a016640 1 -- 172.21.15.28:0/345870111 learned_addr learned my addr 172.21.15.28:0/345870111 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765a817640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76540b42d0 0x7f76540b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46532/0 (socket says 172.21.15.28:46532) 2023-12-16T21:58:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f7659815640 1 --2- 172.21.15.28:0/345870111 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 0x7f765415be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765a817640 1 -- 172.21.15.28:0/345870111 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 msgr2=0x7f765415be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765a817640 1 --2- 172.21.15.28:0/345870111 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 0x7f765415be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765a817640 1 -- 172.21.15.28:0/345870111 >> v1:172.21.15.28:6789/0 conn(0x7f7654158ca0 legacy=0x7f765415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f765a817640 1 -- 172.21.15.28:0/345870111 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765415f6f0 con 0x7f76540b42d0 2023-12-16T21:58:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f7659014640 1 -- 172.21.15.28:0/345870111 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673365908 0 0) 0x7f76540b4c40 con 0x7f7654158ca0 2023-12-16T21:58:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.614+0000 7f7659815640 1 --2- 172.21.15.28:0/345870111 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 0x7f765415be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765a817640 1 --2- 172.21.15.28:0/345870111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76540b4700 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f764c009d90 tx=0x7f764c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b5431829f018 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f7659014640 1 -- 172.21.15.28:0/345870111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c0a5920 con 0x7f76540b42d0 2023-12-16T21:58:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f7659014640 1 -- 172.21.15.28:0/345870111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764c0a5a80 con 0x7f76540b42d0 2023-12-16T21:58:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f7659014640 1 -- 172.21.15.28:0/345870111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c0a5e10 con 0x7f76540b42d0 2023-12-16T21:58:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 -- 172.21.15.28:0/345870111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 msgr2=0x7f76540b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 --2- 172.21.15.28:0/345870111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76540b4700 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f764c009d90 tx=0x7f764c01c950 comp rx=0 tx=0).stop 2023-12-16T21:58:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 -- 172.21.15.28:0/345870111 shutdown_connections 2023-12-16T21:58:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 --2- 172.21.15.28:0/345870111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76540b4700 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 --2- 172.21.15.28:0/345870111 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 0x7f765415be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 -- 172.21.15.28:0/345870111 >> 172.21.15.28:0/345870111 conn(0x7f76540b0cc0 msgr2=0x7f76540b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 -- 172.21.15.28:0/345870111 shutdown_connections 2023-12-16T21:58:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 -- 172.21.15.28:0/345870111 wait complete. 2023-12-16T21:58:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 Processor -- start 2023-12-16T21:58:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 -- start start 2023-12-16T21:58:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76541559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.618+0000 7f765c27a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 0x7f765415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765c27a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7654158ca0 0x7f7654163480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765c27a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7654155f00 con 0x7f76540b42d0 2023-12-16T21:58:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765c27a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76541601a0 con 0x7f7654158ca0 2023-12-16T21:58:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765c27a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7654160480 con 0x7f76540cba10 2023-12-16T21:58:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76541559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76541559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46538/0 (socket says 172.21.15.28:46538) 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 -- 172.21.15.28:0/3944750765 learned_addr learned my addr 172.21.15.28:0/3944750765 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 -- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 msgr2=0x7f765415a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 0x7f765415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 -- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7654158ca0 msgr2=0x7f7654163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7654158ca0 0x7f7654163480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 -- 172.21.15.28:0/3944750765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c009a70 con 0x7f76540b42d0 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765a016640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76541559c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f765000c550 tx=0x7f765000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f764affd640 1 -- 172.21.15.28:0/3944750765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650005cf0 con 0x7f76540b42d0 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f764affd640 1 -- 172.21.15.28:0/3944750765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7650005e70 con 0x7f76540b42d0 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7654160760 con 0x7f76540b42d0 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f764affd640 1 -- 172.21.15.28:0/3944750765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650007310 con 0x7f76540b42d0 2023-12-16T21:58:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765415aa60 con 0x7f76540b42d0 2023-12-16T21:58:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76540693d0 con 0x7f76540b42d0 2023-12-16T21:58:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f764affd640 1 -- 172.21.15.28:0/3944750765 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7650007470 con 0x7f76540b42d0 2023-12-16T21:58:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f764affd640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7638066470 0x7f7638068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f764affd640 1 -- 172.21.15.28:0/3944750765 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7650083490 con 0x7f76540b42d0 2023-12-16T21:58:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f7659815640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7638066470 0x7f7638068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.622+0000 7f7659815640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7638066470 0x7f7638068930 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7644006e90 tx=0x7f7644008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.626+0000 7f764affd640 1 -- 172.21.15.28:0/3944750765 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7650054430 con 0x7f76540b42d0 2023-12-16T21:58:19.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76540b1990 con 0x7f76540b42d0 2023-12-16T21:58:19.788 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f764affd640 1 -- 172.21.15.28:0/3944750765 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7650053fc0 con 0x7f76540b42d0 2023-12-16T21:58:19.788 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7638066470 msgr2=0x7f7638068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7638066470 0x7f7638068930 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7644006e90 tx=0x7f7644008040 comp rx=0 tx=0).stop 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 msgr2=0x7f76541559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76541559c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f765000c550 tx=0x7f765000ca10 comp rx=0 tx=0).stop 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 shutdown_connections 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7638066470 0x7f7638068930 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7654158ca0 0x7f7654163480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76540cba10 0x7f765415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 --2- 172.21.15.28:0/3944750765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76540b42d0 0x7f76541559c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 >> 172.21.15.28:0/3944750765 conn(0x7f76540b0cc0 msgr2=0x7f7654159110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 shutdown_connections 2023-12-16T21:58:19.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.786+0000 7f765c27a640 1 -- 172.21.15.28:0/3944750765 wait complete. 2023-12-16T21:58:19.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8218f84640 1 Processor -- start 2023-12-16T21:58:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8218f84640 1 -- start start 2023-12-16T21:58:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8218f84640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82140aeaa0 0x7f8214160de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8218f84640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f82140ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8218f84640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f821415d990 con 0x7f82140acd80 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8218f84640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8214161400 con 0x7f82140aeaa0 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8218f84640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82141616e0 con 0x7f82140b4770 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f821259c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82140aeaa0 0x7f8214160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f821259c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82140aeaa0 0x7f8214160de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46546/0 (socket says 172.21.15.28:46546) 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f821259c640 1 -- 172.21.15.28:0/373058116 learned_addr learned my addr 172.21.15.28:0/373058116 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8211d9b640 1 --2- 172.21.15.28:0/373058116 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f82140ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f821259c640 1 -- 172.21.15.28:0/373058116 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 msgr2=0x7f82140ac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f821259c640 1 --2- 172.21.15.28:0/373058116 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f82140ac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f821259c640 1 -- 172.21.15.28:0/373058116 >> v1:172.21.15.28:6789/0 conn(0x7f82140acd80 legacy=0x7f821406b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f821259c640 1 -- 172.21.15.28:0/373058116 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82141619c0 con 0x7f82140aeaa0 2023-12-16T21:58:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.930+0000 7f8211d9b640 1 --2- 172.21.15.28:0/373058116 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f82140ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f821259c640 1 --2- 172.21.15.28:0/373058116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f8214160de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8204009e50 tx=0x7f820401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f694cba9eb3eac35 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f821159a640 1 -- 172.21.15.28:0/373058116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040a5830 con 0x7f82140aeaa0 2023-12-16T21:58:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f821159a640 1 -- 172.21.15.28:0/373058116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82040a5990 con 0x7f82140aeaa0 2023-12-16T21:58:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f821159a640 1 -- 172.21.15.28:0/373058116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040a5d20 con 0x7f82140aeaa0 2023-12-16T21:58:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 -- 172.21.15.28:0/373058116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 msgr2=0x7f8214160de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 --2- 172.21.15.28:0/373058116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f8214160de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8204009e50 tx=0x7f820401c950 comp rx=0 tx=0).stop 2023-12-16T21:58:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 -- 172.21.15.28:0/373058116 shutdown_connections 2023-12-16T21:58:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 --2- 172.21.15.28:0/373058116 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f82140ac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 --2- 172.21.15.28:0/373058116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f8214160de0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 -- 172.21.15.28:0/373058116 >> 172.21.15.28:0/373058116 conn(0x7f82140710b0 msgr2=0x7f82140714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 -- 172.21.15.28:0/373058116 shutdown_connections 2023-12-16T21:58:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 -- 172.21.15.28:0/373058116 wait complete. 2023-12-16T21:58:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 Processor -- start 2023-12-16T21:58:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.934+0000 7f8218f84640 1 -- start start 2023-12-16T21:58:19.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82140acd80 0x7f8214156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f821413bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f821415c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82141567f0 con 0x7f82140aeaa0 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8214155960 con 0x7f82140acd80 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821413c700 con 0x7f82140b4770 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f821413bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f821413bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46562/0 (socket says 172.21.15.28:46562) 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 -- 172.21.15.28:0/246494385 learned_addr learned my addr 172.21.15.28:0/246494385 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f821259c640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82140acd80 0x7f8214156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8212d9d640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f821415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 -- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 msgr2=0x7f821415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f821415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 -- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82140acd80 msgr2=0x7f8214156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82140acd80 0x7f8214156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 -- 172.21.15.28:0/246494385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8204009b30 con 0x7f82140aeaa0 2023-12-16T21:58:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8211d9b640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f821413bce0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00c7f0 tx=0x7f81fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f820affd640 1 -- 172.21.15.28:0/246494385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc007820 con 0x7f82140aeaa0 2023-12-16T21:58:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f820affd640 1 -- 172.21.15.28:0/246494385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc007980 con 0x7f82140aeaa0 2023-12-16T21:58:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8214155b70 con 0x7f82140aeaa0 2023-12-16T21:58:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f820affd640 1 -- 172.21.15.28:0/246494385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc007cf0 con 0x7f82140aeaa0 2023-12-16T21:58:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8214156130 con 0x7f82140aeaa0 2023-12-16T21:58:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82140674d0 con 0x7f82140aeaa0 2023-12-16T21:58:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f820affd640 1 -- 172.21.15.28:0/246494385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f81fc0113f0 con 0x7f82140aeaa0 2023-12-16T21:58:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f820affd640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81f4066470 0x7f81f4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f820affd640 1 -- 172.21.15.28:0/246494385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81fc084170 con 0x7f82140aeaa0 2023-12-16T21:58:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.938+0000 7f821259c640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81f4066470 0x7f81f4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.942+0000 7f821259c640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81f4066470 0x7f81f4068930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f82040ae000 tx=0x7f82040a7070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:19.942+0000 7f820affd640 1 -- 172.21.15.28:0/246494385 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f81fc0515f0 con 0x7f82140aeaa0 2023-12-16T21:58:20.096 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f821406dee0 con 0x7f82140aeaa0 2023-12-16T21:58:20.096 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f820affd640 1 -- 172.21.15.28:0/246494385 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f81fc054c10 con 0x7f82140aeaa0 2023-12-16T21:58:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81f4066470 msgr2=0x7f81f4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81f4066470 0x7f81f4068930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f82040ae000 tx=0x7f82040a7070 comp rx=0 tx=0).stop 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 msgr2=0x7f821413bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f821413bce0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00c7f0 tx=0x7f81fc00ccb0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 shutdown_connections 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81f4066470 0x7f81f4068930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f82140b4770 0x7f821415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82140aeaa0 0x7f821413bce0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 --2- 172.21.15.28:0/246494385 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82140acd80 0x7f8214156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 >> 172.21.15.28:0/246494385 conn(0x7f82140710b0 msgr2=0x7f8214153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 shutdown_connections 2023-12-16T21:58:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.094+0000 7f8218f84640 1 -- 172.21.15.28:0/246494385 wait complete. 2023-12-16T21:58:20.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:20.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.238+0000 7fc237efe640 1 Processor -- start 2023-12-16T21:58:20.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.238+0000 7fc237efe640 1 -- start start 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc230158ca0 0x7fc23015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc2300cba10 0x7fc23015be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23015b670 con 0x7fc2300b42d0 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23015f410 con 0x7fc230158ca0 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23015f6f0 con 0x7fc2300cba10 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235c9a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc230158ca0 0x7fc23015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235c9a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc230158ca0 0x7fc23015eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46566/0 (socket says 172.21.15.28:46566) 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235c9a640 1 -- 172.21.15.28:0/3895877482 learned_addr learned my addr 172.21.15.28:0/3895877482 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235499640 1 --2- 172.21.15.28:0/3895877482 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc2300cba10 0x7fc23015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235c9a640 1 -- 172.21.15.28:0/3895877482 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc2300cba10 msgr2=0x7fc23015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235c9a640 1 --2- 172.21.15.28:0/3895877482 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc2300cba10 0x7fc23015be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235c9a640 1 -- 172.21.15.28:0/3895877482 >> v1:172.21.15.122:6789/0 conn(0x7fc2300b42d0 legacy=0x7fc2300703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235c9a640 1 -- 172.21.15.28:0/3895877482 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23015b8e0 con 0x7fc230158ca0 2023-12-16T21:58:20.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235499640 1 --2- 172.21.15.28:0/3895877482 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc2300cba10 0x7fc23015be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc235c9a640 1 --2- 172.21.15.28:0/3895877482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc230158ca0 0x7fc23015eb50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009870 tx=0x7fc22c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8bf16ad8fdb1787 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc234c98640 1 -- 172.21.15.28:0/3895877482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0aa640 con 0x7fc230158ca0 2023-12-16T21:58:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc234c98640 1 -- 172.21.15.28:0/3895877482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c0aa7a0 con 0x7fc230158ca0 2023-12-16T21:58:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc234c98640 1 -- 172.21.15.28:0/3895877482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0aab30 con 0x7fc230158ca0 2023-12-16T21:58:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 -- 172.21.15.28:0/3895877482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc230158ca0 msgr2=0x7fc23015eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 --2- 172.21.15.28:0/3895877482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc230158ca0 0x7fc23015eb50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009870 tx=0x7fc22c01c500 comp rx=0 tx=0).stop 2023-12-16T21:58:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 -- 172.21.15.28:0/3895877482 shutdown_connections 2023-12-16T21:58:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 --2- 172.21.15.28:0/3895877482 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc2300cba10 0x7fc23015be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 --2- 172.21.15.28:0/3895877482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc230158ca0 0x7fc23015eb50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 -- 172.21.15.28:0/3895877482 >> 172.21.15.28:0/3895877482 conn(0x7fc2300b0cc0 msgr2=0x7fc2300b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 -- 172.21.15.28:0/3895877482 shutdown_connections 2023-12-16T21:58:20.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.242+0000 7fc237efe640 1 -- 172.21.15.28:0/3895877482 wait complete. 2023-12-16T21:58:20.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 Processor -- start 2023-12-16T21:58:20.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 -- start start 2023-12-16T21:58:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2300b42d0 0x7fc230149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300cba10 0x7fc230148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc230158ca0 0x7fc230148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2301492c0 con 0x7fc2300cba10 2023-12-16T21:58:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc230149c90 con 0x7fc2300b42d0 2023-12-16T21:58:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc230149f70 con 0x7fc230158ca0 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300cba10 0x7fc230148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300cba10 0x7fc230148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46574/0 (socket says 172.21.15.28:46574) 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 -- 172.21.15.28:0/2403635764 learned_addr learned my addr 172.21.15.28:0/2403635764 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235c9a640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2300b42d0 0x7fc230149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc23649b640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc230158ca0 0x7fc230148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 -- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc230158ca0 msgr2=0x7fc230148310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc230158ca0 0x7fc230148310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 -- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2300b42d0 msgr2=0x7fc230149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2300b42d0 0x7fc230149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 -- 172.21.15.28:0/2403635764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0094d0 con 0x7fc2300cba10 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc23649b640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc230158ca0 0x7fc230148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc235499640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300cba10 0x7fc230148d80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc22000c7f0 tx=0x7fc22000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc2267fc640 1 -- 172.21.15.28:0/2403635764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22000cea0 con 0x7fc2300cba10 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc2267fc640 1 -- 172.21.15.28:0/2403635764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc220007a30 con 0x7fc2300cba10 2023-12-16T21:58:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23014a250 con 0x7fc2300cba10 2023-12-16T21:58:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc2267fc640 1 -- 172.21.15.28:0/2403635764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc220007dc0 con 0x7fc2300cba10 2023-12-16T21:58:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23014a810 con 0x7fc2300cba10 2023-12-16T21:58:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.246+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23014aa30 con 0x7fc2300cba10 2023-12-16T21:58:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.250+0000 7fc2267fc640 1 -- 172.21.15.28:0/2403635764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fc2200113f0 con 0x7fc2300cba10 2023-12-16T21:58:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.250+0000 7fc2267fc640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc2140664a0 0x7fc214068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.250+0000 7fc2267fc640 1 -- 172.21.15.28:0/2403635764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc220084090 con 0x7fc2300cba10 2023-12-16T21:58:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.250+0000 7fc235c9a640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc2140664a0 0x7fc214068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.250+0000 7fc235c9a640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc2140664a0 0x7fc214068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0ae000 tx=0x7fc22c0a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.254+0000 7fc2267fc640 1 -- 172.21.15.28:0/2403635764 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fc220051470 con 0x7fc2300cba10 2023-12-16T21:58:20.412 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.410+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc2300693d0 con 0x7fc2300cba10 2023-12-16T21:58:20.413 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.410+0000 7fc2267fc640 1 -- 172.21.15.28:0/2403635764 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc220054a90 con 0x7fc2300cba10 2023-12-16T21:58:20.413 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:20.415 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.410+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc2140664a0 msgr2=0x7fc214068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.415 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.410+0000 7fc237efe640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc2140664a0 0x7fc214068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0ae000 tx=0x7fc22c0a7000 comp rx=0 tx=0).stop 2023-12-16T21:58:20.415 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.410+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300cba10 msgr2=0x7fc230148d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.410+0000 7fc237efe640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300cba10 0x7fc230148d80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc22000c7f0 tx=0x7fc22000ccb0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.414+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 shutdown_connections 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.414+0000 7fc237efe640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc2140664a0 0x7fc214068960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.414+0000 7fc237efe640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc230158ca0 0x7fc230148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.414+0000 7fc237efe640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300cba10 0x7fc230148d80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.414+0000 7fc237efe640 1 --2- 172.21.15.28:0/2403635764 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2300b42d0 0x7fc230149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.414+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 >> 172.21.15.28:0/2403635764 conn(0x7fc2300b0cc0 msgr2=0x7fc230073460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.414+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 shutdown_connections 2023-12-16T21:58:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.414+0000 7fc237efe640 1 -- 172.21.15.28:0/2403635764 wait complete. 2023-12-16T21:58:20.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:58:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e24639640 1 Processor -- start 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e24639640 1 -- start start 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e24639640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c144840 0x7f0e1c06c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e24639640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e1c072180 0x7f0e1c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e24639640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1c0710e0 con 0x7f0e1c070460 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e24639640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1c15f010 con 0x7f0e1c072180 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e24639640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c0713c0 con 0x7f0e1c144840 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e223d5640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c144840 0x7f0e1c06c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e21bd4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e1c072180 0x7f0e1c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e21bd4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0e1c072180 0x7f0e1c06c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:37442/0 (socket says 172.21.15.28:37442) 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e21bd4640 1 -- 172.21.15.28:0/1612960486 learned_addr learned my addr 172.21.15.28:0/1612960486 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e21bd4640 1 -- 172.21.15.28:0/1612960486 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c144840 msgr2=0x7f0e1c06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e21bd4640 1 --2- 172.21.15.28:0/1612960486 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c144840 0x7f0e1c06c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e21bd4640 1 -- 172.21.15.28:0/1612960486 >> v1:172.21.15.122:6789/0 conn(0x7f0e1c070460 legacy=0x7f0e1c0708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.550+0000 7f0e21bd4640 1 -- 172.21.15.28:0/1612960486 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0716a0 con 0x7f0e1c072180 2023-12-16T21:58:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e21bd4640 1 --2- 172.21.15.28:0/1612960486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 0x7f0e1c06c800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009870 tx=0x7f0e1001c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6bff006cb5263cb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e213d3640 1 -- 172.21.15.28:0/1612960486 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100aa670 con 0x7f0e1c072180 2023-12-16T21:58:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e213d3640 1 -- 172.21.15.28:0/1612960486 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e100aa7d0 con 0x7f0e1c072180 2023-12-16T21:58:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e213d3640 1 -- 172.21.15.28:0/1612960486 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e100aab60 con 0x7f0e1c072180 2023-12-16T21:58:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 -- 172.21.15.28:0/1612960486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 msgr2=0x7f0e1c06c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 --2- 172.21.15.28:0/1612960486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 0x7f0e1c06c800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009870 tx=0x7f0e1001c510 comp rx=0 tx=0).stop 2023-12-16T21:58:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 -- 172.21.15.28:0/1612960486 shutdown_connections 2023-12-16T21:58:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 --2- 172.21.15.28:0/1612960486 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 0x7f0e1c06c800 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 --2- 172.21.15.28:0/1612960486 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c144840 0x7f0e1c06c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 -- 172.21.15.28:0/1612960486 >> 172.21.15.28:0/1612960486 conn(0x7f0e1c06a530 msgr2=0x7f0e1c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 -- 172.21.15.28:0/1612960486 shutdown_connections 2023-12-16T21:58:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 -- 172.21.15.28:0/1612960486 wait complete. 2023-12-16T21:58:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.554+0000 7f0e24639640 1 Processor -- start 2023-12-16T21:58:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 -- start start 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c070460 0x7f0e1c13eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 0x7f0e1c164550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1c144840 0x7f0e1c159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c147430 con 0x7f0e1c144840 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c1499c0 con 0x7f0e1c072180 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c149ca0 con 0x7f0e1c070460 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1c144840 0x7f0e1c159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1c144840 0x7f0e1c159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46582/0 (socket says 172.21.15.28:46582) 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 -- 172.21.15.28:0/859484884 learned_addr learned my addr 172.21.15.28:0/859484884 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e223d5640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c070460 0x7f0e1c13eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e21bd4640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 0x7f0e1c164550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 -- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c070460 msgr2=0x7f0e1c13eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c070460 0x7f0e1c13eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 -- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 msgr2=0x7f0e1c164550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 0x7f0e1c164550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 -- 172.21.15.28:0/859484884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e100094d0 con 0x7f0e1c144840 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e22bd6640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1c144840 0x7f0e1c159e40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c00c430 tx=0x7f0e0c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e1affd640 1 -- 172.21.15.28:0/859484884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c005220 con 0x7f0e1c144840 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e1affd640 1 -- 172.21.15.28:0/859484884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0c005380 con 0x7f0e1c144840 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e1affd640 1 -- 172.21.15.28:0/859484884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c007270 con 0x7f0e1c144840 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1c149f80 con 0x7f0e1c144840 2023-12-16T21:58:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1c13f060 con 0x7f0e1c144840 2023-12-16T21:58:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1c14a430 con 0x7f0e1c144840 2023-12-16T21:58:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e1affd640 1 -- 172.21.15.28:0/859484884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f0e0c0073d0 con 0x7f0e1c144840 2023-12-16T21:58:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e1affd640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e08066470 0x7f0e08068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e1affd640 1 -- 172.21.15.28:0/859484884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e0c0833d0 con 0x7f0e1c144840 2023-12-16T21:58:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.558+0000 7f0e223d5640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e08066470 0x7f0e08068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.562+0000 7f0e223d5640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e08066470 0x7f0e08068930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0e140073f0 tx=0x7f0e1400b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.562+0000 7f0e1affd640 1 -- 172.21.15.28:0/859484884 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f0e0c0508e0 con 0x7f0e1c144840 2023-12-16T21:58:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.730+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0e1c147570 con 0x7f0e1c144840 2023-12-16T21:58:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e1affd640 1 -- 172.21.15.28:0/859484884 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0e0c053f00 con 0x7f0e1c144840 2023-12-16T21:58:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e08066470 msgr2=0x7f0e08068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e08066470 0x7f0e08068930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0e140073f0 tx=0x7f0e1400b040 comp rx=0 tx=0).stop 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1c144840 msgr2=0x7f0e1c159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1c144840 0x7f0e1c159e40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c00c430 tx=0x7f0e0c00c8f0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 shutdown_connections 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0e08066470 0x7f0e08068930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1c144840 0x7f0e1c159e40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0e1c072180 0x7f0e1c164550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 --2- 172.21.15.28:0/859484884 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0e1c070460 0x7f0e1c13eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 >> 172.21.15.28:0/859484884 conn(0x7f0e1c06a530 msgr2=0x7f0e1c165140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 shutdown_connections 2023-12-16T21:58:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:20.734+0000 7f0e24639640 1 -- 172.21.15.28:0/859484884 wait complete. 2023-12-16T21:58:20.755 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2023-12-16T21:58:20.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:20.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-12-16T21:58:20.855 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:58:20.855 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2023-12-16T21:58:20.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:20.855 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-16T21:58:20.866 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:58:20.866 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:20.872 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:58:20.872 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:20.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.090+0000 7f66ed664640 1 Processor -- start 2023-12-16T21:58:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.090+0000 7f66ed664640 1 -- start start 2023-12-16T21:58:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66ed664640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66e806bc50 0x7f66e815da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66ed664640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e815dfa0 0x7f66e80cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66ed664640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e815e790 con 0x7f66e80aea20 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66ed664640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e815a730 con 0x7f66e806bc50 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66ed664640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e815aa10 con 0x7f66e815dfa0 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66dffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66e806bc50 0x7f66e815da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66dffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66e806bc50 0x7f66e815da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46584/0 (socket says 172.21.15.28:46584) 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66dffff640 1 -- 172.21.15.28:0/96759429 learned_addr learned my addr 172.21.15.28:0/96759429 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66e77fe640 1 --2- 172.21.15.28:0/96759429 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e815dfa0 0x7f66e80cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66dffff640 1 -- 172.21.15.28:0/96759429 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e815dfa0 msgr2=0x7f66e80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66dffff640 1 --2- 172.21.15.28:0/96759429 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e815dfa0 0x7f66e80cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66dffff640 1 -- 172.21.15.28:0/96759429 >> v1:172.21.15.122:6789/0 conn(0x7f66e80aea20 legacy=0x7f66e806b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66dffff640 1 -- 172.21.15.28:0/96759429 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e815acf0 con 0x7f66e806bc50 2023-12-16T21:58:21.098 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66e77fe640 1 --2- 172.21.15.28:0/96759429 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e815dfa0 0x7f66e80cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:21.098 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66dffff640 1 --2- 172.21.15.28:0/96759429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e806bc50 0x7f66e815da60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009870 tx=0x7f66d001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2fc7922eefb354 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66e67fc640 1 -- 172.21.15.28:0/96759429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d00ac630 con 0x7f66e806bc50 2023-12-16T21:58:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66e67fc640 1 -- 172.21.15.28:0/96759429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d00ac790 con 0x7f66e806bc50 2023-12-16T21:58:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.094+0000 7f66e67fc640 1 -- 172.21.15.28:0/96759429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d00acb20 con 0x7f66e806bc50 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- 172.21.15.28:0/96759429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e806bc50 msgr2=0x7f66e815da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 --2- 172.21.15.28:0/96759429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e806bc50 0x7f66e815da60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009870 tx=0x7f66d001c500 comp rx=0 tx=0).stop 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- 172.21.15.28:0/96759429 shutdown_connections 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 --2- 172.21.15.28:0/96759429 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e815dfa0 0x7f66e80cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 --2- 172.21.15.28:0/96759429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e806bc50 0x7f66e815da60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- 172.21.15.28:0/96759429 >> 172.21.15.28:0/96759429 conn(0x7f66e80b1a50 msgr2=0x7f66e8070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- 172.21.15.28:0/96759429 shutdown_connections 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- 172.21.15.28:0/96759429 wait complete. 2023-12-16T21:58:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 Processor -- start 2023-12-16T21:58:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- start start 2023-12-16T21:58:21.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e806bc50 0x7f66e81373e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66e80aea20 0x7f66e80b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e815dfa0 0x7f66e80b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e8139450 con 0x7f66e815dfa0 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e806d8a0 con 0x7f66e80aea20 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e8138b20 con 0x7f66e806bc50 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e815dfa0 0x7f66e80b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e815dfa0 0x7f66e80b0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46598/0 (socket says 172.21.15.28:46598) 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 -- 172.21.15.28:0/1604011009 learned_addr learned my addr 172.21.15.28:0/1604011009 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e6ffd640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e806bc50 0x7f66e81373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66dffff640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66e80aea20 0x7f66e80b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 -- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e806bc50 msgr2=0x7f66e81373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e806bc50 0x7f66e81373e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 -- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66e80aea20 msgr2=0x7f66e80b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66e80aea20 0x7f66e80b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 -- 172.21.15.28:0/1604011009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d00094d0 con 0x7f66e815dfa0 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66dffff640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66e80aea20 0x7f66e80b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66e77fe640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e815dfa0 0x7f66e80b0630 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f66e000cd50 tx=0x7f66e000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66df7fe640 1 -- 172.21.15.28:0/1604011009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e000f680 con 0x7f66e815dfa0 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66df7fe640 1 -- 172.21.15.28:0/1604011009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e000f800 con 0x7f66e815dfa0 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66df7fe640 1 -- 172.21.15.28:0/1604011009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e000fbb0 con 0x7f66e815dfa0 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e8138e00 con 0x7f66e815dfa0 2023-12-16T21:58:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.098+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e806cae0 con 0x7f66e815dfa0 2023-12-16T21:58:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.102+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e81395c0 con 0x7f66e815dfa0 2023-12-16T21:58:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.102+0000 7f66df7fe640 1 -- 172.21.15.28:0/1604011009 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f66e000b040 con 0x7f66e815dfa0 2023-12-16T21:58:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.102+0000 7f66df7fe640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66c8066470 0x7f66c8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.102+0000 7f66df7fe640 1 -- 172.21.15.28:0/1604011009 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66e0084800 con 0x7f66e815dfa0 2023-12-16T21:58:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.102+0000 7f66e6ffd640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66c8066470 0x7f66c8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.102+0000 7f66e6ffd640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66c8066470 0x7f66c8068930 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f66d8007fd0 tx=0x7f66d8007e40 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.109 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.106+0000 7f66df7fe640 1 -- 172.21.15.28:0/1604011009 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f66e0051c90 con 0x7f66e815dfa0 2023-12-16T21:58:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.262+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66e80674d0 con 0x7f66e815dfa0 2023-12-16T21:58:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.266+0000 7f66df7fe640 1 -- 172.21.15.28:0/1604011009 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f66e001d090 con 0x7f66e815dfa0 2023-12-16T21:58:21.269 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:21.269 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.266+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66c8066470 msgr2=0x7f66c8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.266+0000 7f66ed664640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66c8066470 0x7f66c8068930 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f66d8007fd0 tx=0x7f66d8007e40 comp rx=0 tx=0).stop 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.266+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e815dfa0 msgr2=0x7f66e80b0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.266+0000 7f66ed664640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e815dfa0 0x7f66e80b0630 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f66e000cd50 tx=0x7f66e000a650 comp rx=0 tx=0).stop 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.270+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 shutdown_connections 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.270+0000 7f66ed664640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f66c8066470 0x7f66c8068930 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.270+0000 7f66ed664640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e815dfa0 0x7f66e80b0630 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.270+0000 7f66ed664640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f66e80aea20 0x7f66e80b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.270+0000 7f66ed664640 1 --2- 172.21.15.28:0/1604011009 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f66e806bc50 0x7f66e81373e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.270+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 >> 172.21.15.28:0/1604011009 conn(0x7f66e80b1a50 msgr2=0x7f66e81589a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.270+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 shutdown_connections 2023-12-16T21:58:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.270+0000 7f66ed664640 1 -- 172.21.15.28:0/1604011009 wait complete. 2023-12-16T21:58:21.290 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:21.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc513ac640 1 Processor -- start 2023-12-16T21:58:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc513ac640 1 -- start start 2023-12-16T21:58:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc513ac640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efc4c146760 0x7efc4c06f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc513ac640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c06f940 con 0x7efc4c143a10 2023-12-16T21:58:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc513ac640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c161030 con 0x7efc4c06ac40 2023-12-16T21:58:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc513ac640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4c1619d0 con 0x7efc4c146760 2023-12-16T21:58:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4affd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efc4c143a10 0x7efc4c06a530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47476/0 (socket says 172.21.15.28:47476) 2023-12-16T21:58:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4affd640 1 -- 172.21.15.28:0/563848348 learned_addr learned my addr 172.21.15.28:0/563848348 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/563848348 >> v2:172.21.15.122:3300/0 conn(0x7efc4c146760 0x7efc4c06f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017206927 0 0) 0x7efc4c06f940 con 0x7efc4c143a10 2023-12-16T21:58:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc34003530 con 0x7efc4c143a10 2023-12-16T21:58:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 583352479 0 0) 0x7efc34003530 con 0x7efc4c143a10 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 >> v2:172.21.15.122:3300/0 conn(0x7efc4c146760 msgr2=0x7efc4c06f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 --2- 172.21.15.28:0/563848348 >> v2:172.21.15.122:3300/0 conn(0x7efc4c146760 0x7efc4c06f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 >> v1:172.21.15.122:6789/0 conn(0x7efc4c06ac40 legacy=0x7efc4c0b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c161310 con 0x7efc4c143a10 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7efc3c001680 con 0x7efc4c143a10 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 >> v1:172.21.15.28:6789/0 conn(0x7efc4c143a10 legacy=0x7efc4c06a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc34003980 0x7efc34003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc340047d0 0x7efc34004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc3400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c1619d0 con 0x7efc34005190 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3400c130 con 0x7efc34003980 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3400c2f0 con 0x7efc340047d0 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc3400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4affd640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc34003980 0x7efc34003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4a7fc640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc340047d0 0x7efc34004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4b7fe640 1 -- 172.21.15.28:0/563848348 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc340047d0 msgr2=0x7efc34004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc340047d0 0x7efc34004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4b7fe640 1 -- 172.21.15.28:0/563848348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc34003980 msgr2=0x7efc34003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc34003980 0x7efc34003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.418+0000 7efc4b7fe640 1 -- 172.21.15.28:0/563848348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc34003530 con 0x7efc34005190 2023-12-16T21:58:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc4b7fe640 1 -- 172.21.15.28:0/563848348 reap_dead start 2023-12-16T21:58:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc3400b990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efc44009fc0 tx=0x7efc4401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc59b0e7577667cf server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc440a5bb0 con 0x7efc34005190 2023-12-16T21:58:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4400f040 con 0x7efc34005190 2023-12-16T21:58:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc49ffb640 1 -- 172.21.15.28:0/563848348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc440039a0 con 0x7efc34005190 2023-12-16T21:58:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc513ac640 1 -- 172.21.15.28:0/563848348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 msgr2=0x7efc3400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc513ac640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc3400b990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efc44009fc0 tx=0x7efc4401cca0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc513ac640 1 -- 172.21.15.28:0/563848348 shutdown_connections 2023-12-16T21:58:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc513ac640 1 --2- 172.21.15.28:0/563848348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc3400b990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc513ac640 1 -- 172.21.15.28:0/563848348 >> 172.21.15.28:0/563848348 conn(0x7efc4c0b1db0 msgr2=0x7efc4c0b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc513ac640 1 -- 172.21.15.28:0/563848348 shutdown_connections 2023-12-16T21:58:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.422+0000 7efc513ac640 1 -- 172.21.15.28:0/563848348 wait complete. 2023-12-16T21:58:21.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 Processor -- start 2023-12-16T21:58:21.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 -- start start 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc4c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc4c13b960 0x7efc4c1f8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc4c13bde0 0x7efc4c1f8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c13b540 con 0x7efc34005190 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c1f7300 con 0x7efc4c13bde0 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c0ce440 con 0x7efc4c13b960 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc4c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc4c15c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46618/0 (socket says 172.21.15.28:46618) 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 -- 172.21.15.28:0/2418163840 learned_addr learned my addr 172.21.15.28:0/2418163840 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 -- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc4c13b960 msgr2=0x7efc4c1f8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4a7fc640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc4c13b960 0x7efc4c1f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc4c13b960 0x7efc4c1f8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc4c13bde0 0x7efc4c1f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 -- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc4c13bde0 msgr2=0x7efc4c1f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc4c13bde0 0x7efc4c1f8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 -- 172.21.15.28:0/2418163840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c0ce720 con 0x7efc34005190 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc4affd640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc4c15c940 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efc3c012490 tx=0x7efc3c012950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc337fe640 1 -- 172.21.15.28:0/2418163840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c014810 con 0x7efc34005190 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc337fe640 1 -- 172.21.15.28:0/2418163840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c014970 con 0x7efc34005190 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4c13ae80 con 0x7efc34005190 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc337fe640 1 -- 172.21.15.28:0/2418163840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c014d00 con 0x7efc34005190 2023-12-16T21:58:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4c15ce80 con 0x7efc34005190 2023-12-16T21:58:21.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.426+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4c13ace0 con 0x7efc34005190 2023-12-16T21:58:21.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.430+0000 7efc337fe640 1 -- 172.21.15.28:0/2418163840 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7efc3c0153e0 con 0x7efc34005190 2023-12-16T21:58:21.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.430+0000 7efc337fe640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efc380664a0 0x7efc38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.430+0000 7efc337fe640 1 -- 172.21.15.28:0/2418163840 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc3c0867b0 con 0x7efc34005190 2023-12-16T21:58:21.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.430+0000 7efc4a7fc640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efc380664a0 0x7efc38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.430+0000 7efc4a7fc640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efc380664a0 0x7efc38068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc4c13aad0 tx=0x7efc40009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.434+0000 7efc337fe640 1 -- 172.21.15.28:0/2418163840 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7efc4c13ace0 con 0x7efc34005190 2023-12-16T21:58:21.593 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.590+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc4c13b680 con 0x7efc34005190 2023-12-16T21:58:21.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.590+0000 7efc337fe640 1 -- 172.21.15.28:0/2418163840 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc4c13b680 con 0x7efc34005190 2023-12-16T21:58:21.594 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efc380664a0 msgr2=0x7efc38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efc380664a0 0x7efc38068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc4c13aad0 tx=0x7efc40009040 comp rx=0 tx=0).stop 2023-12-16T21:58:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 msgr2=0x7efc4c15c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc4c15c940 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efc3c012490 tx=0x7efc3c012950 comp rx=0 tx=0).stop 2023-12-16T21:58:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 shutdown_connections 2023-12-16T21:58:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efc380664a0 0x7efc38068960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efc4c13bde0 0x7efc4c1f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efc4c13b960 0x7efc4c1f8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 --2- 172.21.15.28:0/2418163840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc34005190 0x7efc4c15c940 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 >> 172.21.15.28:0/2418163840 conn(0x7efc4c0b1db0 msgr2=0x7efc4c160350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 shutdown_connections 2023-12-16T21:58:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.594+0000 7efc513ac640 1 -- 172.21.15.28:0/2418163840 wait complete. 2023-12-16T21:58:21.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb816e640 1 Processor -- start 2023-12-16T21:58:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb816e640 1 -- start start 2023-12-16T21:58:21.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb816e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbcb015d690 0x7fbcb00cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb816e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb00b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb816e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb00d0540 con 0x7fbcb015d2b0 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb816e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb00aff30 con 0x7fbcb015d690 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb816e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00d1ec0 con 0x7fbcb00d0120 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb5709640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbcb015d690 0x7fbcb00cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb5709640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbcb015d690 0x7fbcb00cfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46626/0 (socket says 172.21.15.28:46626) 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb5709640 1 -- 172.21.15.28:0/1048410357 learned_addr learned my addr 172.21.15.28:0/1048410357 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb670b640 1 --2- 172.21.15.28:0/1048410357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb00b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb5709640 1 -- 172.21.15.28:0/1048410357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 msgr2=0x7fbcb00b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb5709640 1 --2- 172.21.15.28:0/1048410357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb00b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb5709640 1 -- 172.21.15.28:0/1048410357 >> v1:172.21.15.122:6789/0 conn(0x7fbcb015d2b0 legacy=0x7fbcb00d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb5709640 1 -- 172.21.15.28:0/1048410357 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb00b01d0 con 0x7fbcb015d690 2023-12-16T21:58:21.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.742+0000 7fbcb670b640 1 --2- 172.21.15.28:0/1048410357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb00b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:21.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb5709640 1 --2- 172.21.15.28:0/1048410357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d690 0x7fbcb00cfbe0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009870 tx=0x7fbca001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf7500aed7530bb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb4f08640 1 -- 172.21.15.28:0/1048410357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca00aa670 con 0x7fbcb015d690 2023-12-16T21:58:21.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb4f08640 1 -- 172.21.15.28:0/1048410357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca00aa7d0 con 0x7fbcb015d690 2023-12-16T21:58:21.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb4f08640 1 -- 172.21.15.28:0/1048410357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca00aab60 con 0x7fbcb015d690 2023-12-16T21:58:21.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 -- 172.21.15.28:0/1048410357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d690 msgr2=0x7fbcb00cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 --2- 172.21.15.28:0/1048410357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d690 0x7fbcb00cfbe0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009870 tx=0x7fbca001c510 comp rx=0 tx=0).stop 2023-12-16T21:58:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 -- 172.21.15.28:0/1048410357 shutdown_connections 2023-12-16T21:58:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 --2- 172.21.15.28:0/1048410357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb00b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 --2- 172.21.15.28:0/1048410357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d690 0x7fbcb00cfbe0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 -- 172.21.15.28:0/1048410357 >> 172.21.15.28:0/1048410357 conn(0x7fbcb009c510 msgr2=0x7fbcb00d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 -- 172.21.15.28:0/1048410357 shutdown_connections 2023-12-16T21:58:21.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 -- 172.21.15.28:0/1048410357 wait complete. 2023-12-16T21:58:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.746+0000 7fbcb816e640 1 Processor -- start 2023-12-16T21:58:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 -- start start 2023-12-16T21:58:21.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb01536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d2b0 0x7fbcb00c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcb015d690 0x7fbcb00c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb01551b0 con 0x7fbcb015d2b0 2023-12-16T21:58:21.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb0155420 con 0x7fbcb015d690 2023-12-16T21:58:21.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb00cc810 con 0x7fbcb00d0120 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d2b0 0x7fbcb00c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d2b0 0x7fbcb00c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46634/0 (socket says 172.21.15.28:46634) 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 -- 172.21.15.28:0/245789745 learned_addr learned my addr 172.21.15.28:0/245789745 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb670b640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcb015d690 0x7fbcb00c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 -- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 msgr2=0x7fbcb01536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5f0a640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb01536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb01536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 -- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcb015d690 msgr2=0x7fbcb00c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcb015d690 0x7fbcb00c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 -- 172.21.15.28:0/245789745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca00094d0 con 0x7fbcb015d2b0 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5709640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d2b0 0x7fbcb00c8d00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbca001a140 tx=0x7fbca001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbca67fc640 1 -- 172.21.15.28:0/245789745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca00b3800 con 0x7fbcb015d2b0 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbca67fc640 1 -- 172.21.15.28:0/245789745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca00b3980 con 0x7fbcb015d2b0 2023-12-16T21:58:21.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb00ccaf0 con 0x7fbcb015d2b0 2023-12-16T21:58:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbca67fc640 1 -- 172.21.15.28:0/245789745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca00b3d30 con 0x7fbcb015d2b0 2023-12-16T21:58:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb00cc500 con 0x7fbcb015d2b0 2023-12-16T21:58:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb00679f0 con 0x7fbcb015d2b0 2023-12-16T21:58:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbca67fc640 1 -- 172.21.15.28:0/245789745 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fbca00c6390 con 0x7fbcb015d2b0 2023-12-16T21:58:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbca67fc640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fbc940664a0 0x7fbc94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbca67fc640 1 -- 172.21.15.28:0/245789745 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbca011ea50 con 0x7fbcb015d2b0 2023-12-16T21:58:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.750+0000 7fbcb5f0a640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fbc940664a0 0x7fbc94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.754+0000 7fbcb5f0a640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fbc940664a0 0x7fbc94068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbcac000f70 tx=0x7fbcac009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:21.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.754+0000 7fbca67fc640 1 -- 172.21.15.28:0/245789745 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fbca00ebf60 con 0x7fbcb015d2b0 2023-12-16T21:58:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.914+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbcb00693d0 con 0x7fbcb015d2b0 2023-12-16T21:58:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.914+0000 7fbca67fc640 1 -- 172.21.15.28:0/245789745 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbca00ef580 con 0x7fbcb015d2b0 2023-12-16T21:58:21.918 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fbc940664a0 msgr2=0x7fbc94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fbc940664a0 0x7fbc94068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbcac000f70 tx=0x7fbcac009040 comp rx=0 tx=0).stop 2023-12-16T21:58:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d2b0 msgr2=0x7fbcb00c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d2b0 0x7fbcb00c8d00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbca001a140 tx=0x7fbca001ad50 comp rx=0 tx=0).stop 2023-12-16T21:58:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 shutdown_connections 2023-12-16T21:58:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fbc940664a0 0x7fbc94068960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbcb015d690 0x7fbcb00c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb015d2b0 0x7fbcb00c8d00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 --2- 172.21.15.28:0/245789745 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fbcb00d0120 0x7fbcb01536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 >> 172.21.15.28:0/245789745 conn(0x7fbcb009c510 msgr2=0x7fbcb0155f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 shutdown_connections 2023-12-16T21:58:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:21.918+0000 7fbcb816e640 1 -- 172.21.15.28:0/245789745 wait complete. 2023-12-16T21:58:21.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.062+0000 7f62f712b640 1 Processor -- start 2023-12-16T21:58:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.062+0000 7f62f712b640 1 -- start start 2023-12-16T21:58:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f712b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f62f0158ca0 0x7f62f015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f712b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f00cba10 0x7f62f015be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f712b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62f00b42d0 0x7f62f00b4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f712b640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f00b4c40 con 0x7f62f00b42d0 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f712b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f015b670 con 0x7f62f0158ca0 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f712b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f015f410 con 0x7f62f00cba10 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62f00b42d0 0x7f62f00b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62f00b42d0 0x7f62f00b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46640/0 (socket says 172.21.15.28:46640) 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 -- 172.21.15.28:0/3814902147 learned_addr learned my addr 172.21.15.28:0/3814902147 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62effff640 1 --2- 172.21.15.28:0/3814902147 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f00cba10 0x7f62f015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f4ec7640 1 --2- 172.21.15.28:0/3814902147 >> v2:172.21.15.122:3300/0 conn(0x7f62f0158ca0 0x7f62f015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 -- 172.21.15.28:0/3814902147 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f00cba10 msgr2=0x7f62f015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 --2- 172.21.15.28:0/3814902147 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f00cba10 0x7f62f015be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 -- 172.21.15.28:0/3814902147 >> v2:172.21.15.122:3300/0 conn(0x7f62f0158ca0 msgr2=0x7f62f015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 --2- 172.21.15.28:0/3814902147 >> v2:172.21.15.122:3300/0 conn(0x7f62f0158ca0 0x7f62f015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 -- 172.21.15.28:0/3814902147 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f015f6f0 con 0x7f62f00b42d0 2023-12-16T21:58:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62f56c8640 1 --2- 172.21.15.28:0/3814902147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00b42d0 0x7f62f00b4700 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f62f0067f70 tx=0x7f62e801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0ee5ad71d99dd7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62ef7fe640 1 -- 172.21.15.28:0/3814902147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e80aa610 con 0x7f62f00b42d0 2023-12-16T21:58:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62ef7fe640 1 -- 172.21.15.28:0/3814902147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e80a6040 con 0x7f62f00b42d0 2023-12-16T21:58:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.066+0000 7f62ef7fe640 1 -- 172.21.15.28:0/3814902147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e80aa990 con 0x7f62f00b42d0 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- 172.21.15.28:0/3814902147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00b42d0 msgr2=0x7f62f00b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 --2- 172.21.15.28:0/3814902147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00b42d0 0x7f62f00b4700 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f62f0067f70 tx=0x7f62e801c4d0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- 172.21.15.28:0/3814902147 shutdown_connections 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 --2- 172.21.15.28:0/3814902147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00b42d0 0x7f62f00b4700 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 --2- 172.21.15.28:0/3814902147 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f00cba10 0x7f62f015be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 --2- 172.21.15.28:0/3814902147 >> v2:172.21.15.122:3300/0 conn(0x7f62f0158ca0 0x7f62f015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- 172.21.15.28:0/3814902147 >> 172.21.15.28:0/3814902147 conn(0x7f62f00b0cc0 msgr2=0x7f62f00b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- 172.21.15.28:0/3814902147 shutdown_connections 2023-12-16T21:58:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- 172.21.15.28:0/3814902147 wait complete. 2023-12-16T21:58:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 Processor -- start 2023-12-16T21:58:22.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- start start 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62f00b42d0 0x7f62f014a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00cba10 0x7f62f0155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f0158ca0 0x7f62f0155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f0153e30 con 0x7f62f00cba10 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f0152ed0 con 0x7f62f00b42d0 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f712b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f01531b0 con 0x7f62f0158ca0 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00cba10 0x7f62f0155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00cba10 0x7f62f0155230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46646/0 (socket says 172.21.15.28:46646) 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62effff640 1 -- 172.21.15.28:0/1845107529 learned_addr learned my addr 172.21.15.28:0/1845107529 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f4ec7640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62f00b42d0 0x7f62f014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.070+0000 7f62f56c8640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f0158ca0 0x7f62f0155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62effff640 1 -- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f0158ca0 msgr2=0x7f62f0155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62effff640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f0158ca0 0x7f62f0155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62effff640 1 -- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62f00b42d0 msgr2=0x7f62f014a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62effff640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62f00b42d0 0x7f62f014a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62effff640 1 -- 172.21.15.28:0/1845107529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e80094d0 con 0x7f62f00cba10 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62f4ec7640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62f00b42d0 0x7f62f014a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62effff640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00cba10 0x7f62f0155230 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d7d0 tx=0x7f62e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62ed7fa640 1 -- 172.21.15.28:0/1845107529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400b6c0 con 0x7f62f00cba10 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62ed7fa640 1 -- 172.21.15.28:0/1845107529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e400b820 con 0x7f62f00cba10 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f0153490 con 0x7f62f00cba10 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62ed7fa640 1 -- 172.21.15.28:0/1845107529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400bbd0 con 0x7f62f00cba10 2023-12-16T21:58:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f0153a50 con 0x7f62f00cba10 2023-12-16T21:58:22.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f00cc4c0 con 0x7f62f00cba10 2023-12-16T21:58:22.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62ed7fa640 1 -- 172.21.15.28:0/1845107529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f62e400bd30 con 0x7f62f00cba10 2023-12-16T21:58:22.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62ed7fa640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62dc066470 0x7f62dc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62ed7fa640 1 -- 172.21.15.28:0/1845107529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62e40843a0 con 0x7f62f00cba10 2023-12-16T21:58:22.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62f4ec7640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62dc066470 0x7f62dc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.074+0000 7f62f4ec7640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62dc066470 0x7f62dc068930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f62e0003fd0 tx=0x7f62e00039a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.078+0000 7f62ed7fa640 1 -- 172.21.15.28:0/1845107529 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f62e4051820 con 0x7f62f00cba10 2023-12-16T21:58:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.226+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f62f00693d0 con 0x7f62f00cba10 2023-12-16T21:58:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.226+0000 7f62ed7fa640 1 -- 172.21.15.28:0/1845107529 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62e4054e40 con 0x7f62f00cba10 2023-12-16T21:58:22.230 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62dc066470 msgr2=0x7f62dc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62dc066470 0x7f62dc068930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f62e0003fd0 tx=0x7f62e00039a0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00cba10 msgr2=0x7f62f0155230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00cba10 0x7f62f0155230 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d7d0 tx=0x7f62e400dc90 comp rx=0 tx=0).stop 2023-12-16T21:58:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 shutdown_connections 2023-12-16T21:58:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f62dc066470 0x7f62dc068930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f62f0158ca0 0x7f62f0155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f00cba10 0x7f62f0155230 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 --2- 172.21.15.28:0/1845107529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f62f00b42d0 0x7f62f014a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 >> 172.21.15.28:0/1845107529 conn(0x7f62f00b0cc0 msgr2=0x7f62f00aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 shutdown_connections 2023-12-16T21:58:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.230+0000 7f62f712b640 1 -- 172.21.15.28:0/1845107529 wait complete. 2023-12-16T21:58:22.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:58:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f2581a12640 1 Processor -- start 2023-12-16T21:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f2581a12640 1 -- start start 2023-12-16T21:58:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f2581a12640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f257c0cf510 0x7f257c0b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f2581a12640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c15ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f2581a12640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0cc320 con 0x7f257c15b230 2023-12-16T21:58:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f2581a12640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c0cf9a0 con 0x7f257c0cf510 2023-12-16T21:58:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f2581a12640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0cfbf0 con 0x7f257c0b0bb0 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f257a7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f257c0cf510 0x7f257c0b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f257affd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f257c15b230 0x7f257c0cc7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:53050/0 (socket says 172.21.15.28:53050) 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f257b7fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c15ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f257affd640 1 -- 172.21.15.28:0/656631019 learned_addr learned my addr 172.21.15.28:0/656631019 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f257a7fc640 1 -- 172.21.15.28:0/656631019 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 msgr2=0x7f257c15ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f257a7fc640 1 --2- 172.21.15.28:0/656631019 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c15ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f257a7fc640 1 -- 172.21.15.28:0/656631019 >> v1:172.21.15.122:6789/0 conn(0x7f257c15b230 legacy=0x7f257c0cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f2579ffb640 1 -- 172.21.15.28:0/656631019 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565947301 0 0) 0x7f257c0cc320 con 0x7f257c15b230 2023-12-16T21:58:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.370+0000 7f257a7fc640 1 -- 172.21.15.28:0/656631019 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0cbcd0 con 0x7f257c0cf510 2023-12-16T21:58:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f257a7fc640 1 --2- 172.21.15.28:0/656631019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 0x7f257c0b0670 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f256c0097f0 tx=0x7f256c01c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d4b552a445d268d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2579ffb640 1 -- 172.21.15.28:0/656631019 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c0ab600 con 0x7f257c0cf510 2023-12-16T21:58:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2579ffb640 1 -- 172.21.15.28:0/656631019 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c0a7050 con 0x7f257c0cf510 2023-12-16T21:58:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2579ffb640 1 -- 172.21.15.28:0/656631019 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c0ab980 con 0x7f257c0cf510 2023-12-16T21:58:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2581a12640 1 -- 172.21.15.28:0/656631019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 msgr2=0x7f257c0b0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2581a12640 1 --2- 172.21.15.28:0/656631019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 0x7f257c0b0670 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f256c0097f0 tx=0x7f256c01c3f0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2581a12640 1 -- 172.21.15.28:0/656631019 shutdown_connections 2023-12-16T21:58:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2581a12640 1 --2- 172.21.15.28:0/656631019 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c15ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2581a12640 1 --2- 172.21.15.28:0/656631019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 0x7f257c0b0670 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2581a12640 1 -- 172.21.15.28:0/656631019 >> 172.21.15.28:0/656631019 conn(0x7f257c0d0f30 msgr2=0x7f257c0d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2581a12640 1 -- 172.21.15.28:0/656631019 shutdown_connections 2023-12-16T21:58:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.374+0000 7f2581a12640 1 -- 172.21.15.28:0/656631019 wait complete. 2023-12-16T21:58:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 Processor -- start 2023-12-16T21:58:22.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 -- start start 2023-12-16T21:58:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c0c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 0x7f257c09a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f257c15b230 0x7f257c14b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c14bdb0 con 0x7f257c15b230 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0c98f0 con 0x7f257c0cf510 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0c9bd0 con 0x7f257c0b0bb0 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f257c15b230 0x7f257c14b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f257c15b230 0x7f257c14b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46650/0 (socket says 172.21.15.28:46650) 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 -- 172.21.15.28:0/3573617295 learned_addr learned my addr 172.21.15.28:0/3573617295 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257a7fc640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 0x7f257c09a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257affd640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c0c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 -- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 msgr2=0x7f257c0c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c0c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 -- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 msgr2=0x7f257c09a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 0x7f257c09a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 -- 172.21.15.28:0/3573617295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c009420 con 0x7f257c15b230 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257a7fc640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 0x7f257c09a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257affd640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f257b7fe640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f257c15b230 0x7f257c14b200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f257400efc0 tx=0x7f257400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2563fff640 1 -- 172.21.15.28:0/3573617295 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2574009ba0 con 0x7f257c15b230 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2563fff640 1 -- 172.21.15.28:0/3573617295 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257400f040 con 0x7f257c15b230 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c0c9eb0 con 0x7f257c15b230 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2563fff640 1 -- 172.21.15.28:0/3573617295 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25740146c0 con 0x7f257c15b230 2023-12-16T21:58:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c0c94e0 con 0x7f257c15b230 2023-12-16T21:58:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c0d0950 con 0x7f257c15b230 2023-12-16T21:58:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.378+0000 7f2563fff640 1 -- 172.21.15.28:0/3573617295 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f2574014820 con 0x7f257c15b230 2023-12-16T21:58:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.382+0000 7f2563fff640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f25680664a0 0x7f2568068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.382+0000 7f2563fff640 1 -- 172.21.15.28:0/3573617295 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2574084b70 con 0x7f257c15b230 2023-12-16T21:58:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.382+0000 7f257affd640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f25680664a0 0x7f2568068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.382+0000 7f257affd640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f25680664a0 0x7f2568068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2570000f70 tx=0x7f2570009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.386+0000 7f2563fff640 1 -- 172.21.15.28:0/3573617295 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f2574052080 con 0x7f257c15b230 2023-12-16T21:58:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f257c14bef0 con 0x7f257c15b230 2023-12-16T21:58:22.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2563fff640 1 -- 172.21.15.28:0/3573617295 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f257c14bef0 con 0x7f257c15b230 2023-12-16T21:58:22.556 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f25680664a0 msgr2=0x7f2568068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f25680664a0 0x7f2568068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2570000f70 tx=0x7f2570009040 comp rx=0 tx=0).stop 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f257c15b230 msgr2=0x7f257c14b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f257c15b230 0x7f257c14b200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f257400efc0 tx=0x7f257400c490 comp rx=0 tx=0).stop 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 shutdown_connections 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f25680664a0 0x7f2568068960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f257c15b230 0x7f257c14b200 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f257c0cf510 0x7f257c09a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 --2- 172.21.15.28:0/3573617295 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f257c0b0bb0 0x7f257c0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 >> 172.21.15.28:0/3573617295 conn(0x7f257c0d0f30 msgr2=0x7f257c06f720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 shutdown_connections 2023-12-16T21:58:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.554+0000 7f2581a12640 1 -- 172.21.15.28:0/3573617295 wait complete. 2023-12-16T21:58:22.575 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:58:22.575 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:22.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.738+0000 7f19f07d2640 1 Processor -- start 2023-12-16T21:58:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.738+0000 7f19f07d2640 1 -- start start 2023-12-16T21:58:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19f07d2640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e8155860 0x7f19e815c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19f07d2640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f19e806a530 0x7f19e806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19f07d2640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e806d300 con 0x7f19e815ebf0 2023-12-16T21:58:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19f07d2640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e806af00 con 0x7f19e806a530 2023-12-16T21:58:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19f07d2640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e815c5c0 con 0x7f19e8155860 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19eed6f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f19e806a530 0x7f19e806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19ee56e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e8155860 0x7f19e815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19edd6d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f19e815ebf0 0x7f19e80ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:53058/0 (socket says 172.21.15.28:53058) 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19edd6d640 1 -- 172.21.15.28:0/1249553560 learned_addr learned my addr 172.21.15.28:0/1249553560 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19eed6f640 1 -- 172.21.15.28:0/1249553560 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e8155860 msgr2=0x7f19e815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19eed6f640 1 --2- 172.21.15.28:0/1249553560 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e8155860 0x7f19e815c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19eed6f640 1 -- 172.21.15.28:0/1249553560 >> v1:172.21.15.122:6789/0 conn(0x7f19e815ebf0 legacy=0x7f19e80ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19eed6f640 1 -- 172.21.15.28:0/1249553560 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e80ceca0 con 0x7f19e806a530 2023-12-16T21:58:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.742+0000 7f19ee56e640 1 --2- 172.21.15.28:0/1249553560 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e8155860 0x7f19e815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:58:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.746+0000 7f19eed6f640 1 --2- 172.21.15.28:0/1249553560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e806a530 0x7f19e806a9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009d90 tx=0x7f19e001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2fef1d5f375d2c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19ed56c640 1 -- 172.21.15.28:0/1249553560 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00a5a10 con 0x7f19e806a530 2023-12-16T21:58:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19ed56c640 1 -- 172.21.15.28:0/1249553560 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e00a5b70 con 0x7f19e806a530 2023-12-16T21:58:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19ed56c640 1 -- 172.21.15.28:0/1249553560 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e00ad440 con 0x7f19e806a530 2023-12-16T21:58:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19f07d2640 1 -- 172.21.15.28:0/1249553560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e806a530 msgr2=0x7f19e806a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19f07d2640 1 --2- 172.21.15.28:0/1249553560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e806a530 0x7f19e806a9c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009d90 tx=0x7f19e001c950 comp rx=0 tx=0).stop 2023-12-16T21:58:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19f07d2640 1 -- 172.21.15.28:0/1249553560 shutdown_connections 2023-12-16T21:58:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19f07d2640 1 --2- 172.21.15.28:0/1249553560 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e806a530 0x7f19e806a9c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19f07d2640 1 --2- 172.21.15.28:0/1249553560 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e8155860 0x7f19e815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19f07d2640 1 -- 172.21.15.28:0/1249553560 >> 172.21.15.28:0/1249553560 conn(0x7f19e80cfe90 msgr2=0x7f19e80d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19f07d2640 1 -- 172.21.15.28:0/1249553560 shutdown_connections 2023-12-16T21:58:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.750+0000 7f19f07d2640 1 -- 172.21.15.28:0/1249553560 wait complete. 2023-12-16T21:58:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 Processor -- start 2023-12-16T21:58:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 -- start start 2023-12-16T21:58:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e806a530 0x7f19e8153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8155860 0x7f19e8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e815ebf0 0x7f19e8141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e8154c70 con 0x7f19e8155860 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e8155090 con 0x7f19e815ebf0 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e815a4f0 con 0x7f19e806a530 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8155860 0x7f19e8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8155860 0x7f19e8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46660/0 (socket says 172.21.15.28:46660) 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 -- 172.21.15.28:0/3801301288 learned_addr learned my addr 172.21.15.28:0/3801301288 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19eed6f640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e815ebf0 0x7f19e8141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19ee56e640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e806a530 0x7f19e8153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 -- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e806a530 msgr2=0x7f19e8153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e806a530 0x7f19e8153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 -- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e815ebf0 msgr2=0x7f19e8141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e815ebf0 0x7f19e8141090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 -- 172.21.15.28:0/3801301288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4009420 con 0x7f19e8155860 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19eed6f640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e815ebf0 0x7f19e8141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19edd6d640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8155860 0x7f19e8159fb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c4d0 tx=0x7f19d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19deffd640 1 -- 172.21.15.28:0/3801301288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8005cf0 con 0x7f19e8155860 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19deffd640 1 -- 172.21.15.28:0/3801301288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8005e50 con 0x7f19e8155860 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e0009a70 con 0x7f19e8155860 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19deffd640 1 -- 172.21.15.28:0/3801301288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8007270 con 0x7f19e8155860 2023-12-16T21:58:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e81542d0 con 0x7f19e8155860 2023-12-16T21:58:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e80674d0 con 0x7f19e8155860 2023-12-16T21:58:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19deffd640 1 -- 172.21.15.28:0/3801301288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f19d80073d0 con 0x7f19e8155860 2023-12-16T21:58:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.754+0000 7f19deffd640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f19cc066470 0x7f19cc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.758+0000 7f19deffd640 1 -- 172.21.15.28:0/3801301288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19d8083280 con 0x7f19e8155860 2023-12-16T21:58:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.758+0000 7f19ee56e640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f19cc066470 0x7f19cc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.758+0000 7f19ee56e640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f19cc066470 0x7f19cc068930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009be0 tx=0x7f19e4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.758+0000 7f19deffd640 1 -- 172.21.15.28:0/3801301288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f19d80507b0 con 0x7f19e8155860 2023-12-16T21:58:22.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.914+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19e80cd050 con 0x7f19e8155860 2023-12-16T21:58:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19deffd640 1 -- 172.21.15.28:0/3801301288 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f19d8053dd0 con 0x7f19e8155860 2023-12-16T21:58:22.920 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:22.920 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:57:44.448611+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f19cc066470 msgr2=0x7f19cc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f19cc066470 0x7f19cc068930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009be0 tx=0x7f19e4008040 comp rx=0 tx=0).stop 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8155860 msgr2=0x7f19e8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8155860 0x7f19e8159fb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c4d0 tx=0x7f19d800c990 comp rx=0 tx=0).stop 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 shutdown_connections 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f19cc066470 0x7f19cc068930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19e815ebf0 0x7f19e8141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19e8155860 0x7f19e8159fb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 --2- 172.21.15.28:0/3801301288 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f19e806a530 0x7f19e8153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 >> 172.21.15.28:0/3801301288 conn(0x7f19e80cfe90 msgr2=0x7f19e80d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 shutdown_connections 2023-12-16T21:58:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:22.918+0000 7f19f07d2640 1 -- 172.21.15.28:0/3801301288 wait complete. 2023-12-16T21:58:22.938 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:22.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0ceff640 1 Processor -- start 2023-12-16T21:58:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0ceff640 1 -- start start 2023-12-16T21:58:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0ceff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae0815d2b0 0x7fae080d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0ceff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fae080d02f0 0x7fae080b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0ceff640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae080aff30 con 0x7fae080d1cf0 2023-12-16T21:58:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0ceff640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae080b01d0 con 0x7fae0815d2b0 2023-12-16T21:58:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0ceff640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae080b04b0 con 0x7fae080d02f0 2023-12-16T21:58:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0659c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae0815d2b0 0x7fae080d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fadfdd9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fae080d1cf0 0x7fae080cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47490/0 (socket says 172.21.15.28:47490) 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fadfdd9b640 1 -- 172.21.15.28:0/1174897380 learned_addr learned my addr 172.21.15.28:0/1174897380 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0659c640 1 -- 172.21.15.28:0/1174897380 >> v2:172.21.15.122:3300/0 conn(0x7fae080d02f0 msgr2=0x7fae080b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae06d9d640 1 --2- 172.21.15.28:0/1174897380 >> v2:172.21.15.122:3300/0 conn(0x7fae080d02f0 0x7fae080b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae05d9b640 1 -- 172.21.15.28:0/1174897380 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104252924 0 0) 0x7fae080aff30 con 0x7fae080d1cf0 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0659c640 1 --2- 172.21.15.28:0/1174897380 >> v2:172.21.15.122:3300/0 conn(0x7fae080d02f0 0x7fae080b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0659c640 1 -- 172.21.15.28:0/1174897380 >> v1:172.21.15.28:6789/0 conn(0x7fae080d1cf0 legacy=0x7fae080cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae0659c640 1 -- 172.21.15.28:0/1174897380 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae080b0790 con 0x7fae0815d2b0 2023-12-16T21:58:23.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.058+0000 7fae06d9d640 1 --2- 172.21.15.28:0/1174897380 >> v2:172.21.15.122:3300/0 conn(0x7fae080d02f0 0x7fae080b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0659c640 1 --2- 172.21.15.28:0/1174897380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0815d2b0 0x7fae080d17b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fae080cdb20 tx=0x7fadf801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef427f32306f15b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae05d9b640 1 -- 172.21.15.28:0/1174897380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf80ab600 con 0x7fae0815d2b0 2023-12-16T21:58:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae05d9b640 1 -- 172.21.15.28:0/1174897380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf80ab760 con 0x7fae0815d2b0 2023-12-16T21:58:23.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae05d9b640 1 -- 172.21.15.28:0/1174897380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf80abaf0 con 0x7fae0815d2b0 2023-12-16T21:58:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- 172.21.15.28:0/1174897380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0815d2b0 msgr2=0x7fae080d17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1174897380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0815d2b0 0x7fae080d17b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fae080cdb20 tx=0x7fadf801c460 comp rx=0 tx=0).stop 2023-12-16T21:58:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- 172.21.15.28:0/1174897380 shutdown_connections 2023-12-16T21:58:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1174897380 >> v2:172.21.15.122:3300/0 conn(0x7fae080d02f0 0x7fae080b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1174897380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0815d2b0 0x7fae080d17b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- 172.21.15.28:0/1174897380 >> 172.21.15.28:0/1174897380 conn(0x7fae0809c510 msgr2=0x7fae080d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:23.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- 172.21.15.28:0/1174897380 shutdown_connections 2023-12-16T21:58:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- 172.21.15.28:0/1174897380 wait complete. 2023-12-16T21:58:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 Processor -- start 2023-12-16T21:58:23.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- start start 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae080d02f0 0x7fae0815a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae080d1cf0 0x7fae08155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae0815d2b0 0x7fae0814e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0806b380 con 0x7fae080d1cf0 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae08150660 con 0x7fae080d02f0 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0815c080 con 0x7fae0815d2b0 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae080d1cf0 0x7fae08155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae080d1cf0 0x7fae08155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46670/0 (socket says 172.21.15.28:46670) 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 -- 172.21.15.28:0/1680461538 learned_addr learned my addr 172.21.15.28:0/1680461538 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae06d9d640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae0815d2b0 0x7fae0814e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0659c640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae080d02f0 0x7fae0815a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 -- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae0815d2b0 msgr2=0x7fae0814e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae0815d2b0 0x7fae0814e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 -- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae080d02f0 msgr2=0x7fae0815a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae080d02f0 0x7fae0815a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 -- 172.21.15.28:0/1680461538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf8009420 con 0x7fae080d1cf0 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadfdd9b640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae080d1cf0 0x7fae08155910 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fadf00011f0 tx=0x7fadf000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadff7fe640 1 -- 172.21.15.28:0/1680461538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf00074e0 con 0x7fae080d1cf0 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadff7fe640 1 -- 172.21.15.28:0/1680461538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf0007640 con 0x7fae080d1cf0 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0815c360 con 0x7fae080d1cf0 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fadff7fe640 1 -- 172.21.15.28:0/1680461538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0004a10 con 0x7fae080d1cf0 2023-12-16T21:58:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.062+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0815a9a0 con 0x7fae080d1cf0 2023-12-16T21:58:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.066+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae080679f0 con 0x7fae080d1cf0 2023-12-16T21:58:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.066+0000 7fadff7fe640 1 -- 172.21.15.28:0/1680461538 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fadf0004b70 con 0x7fae080d1cf0 2023-12-16T21:58:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.066+0000 7fadff7fe640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fade8066470 0x7fade8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.066+0000 7fadff7fe640 1 -- 172.21.15.28:0/1680461538 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadf00834b0 con 0x7fae080d1cf0 2023-12-16T21:58:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.066+0000 7fae0659c640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fade8066470 0x7fade8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.066+0000 7fae0659c640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fade8066470 0x7fade8068930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fadf80af000 tx=0x7fadf80a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.070+0000 7fadff7fe640 1 -- 172.21.15.28:0/1680461538 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fadf0050890 con 0x7fae080d1cf0 2023-12-16T21:58:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.226+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fae0806b4c0 con 0x7fae080d1cf0 2023-12-16T21:58:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.226+0000 7fadff7fe640 1 -- 172.21.15.28:0/1680461538 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fae0806b4c0 con 0x7fae080d1cf0 2023-12-16T21:58:23.231 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fade8066470 msgr2=0x7fade8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fade8066470 0x7fade8068930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fadf80af000 tx=0x7fadf80a7000 comp rx=0 tx=0).stop 2023-12-16T21:58:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae080d1cf0 msgr2=0x7fae08155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae080d1cf0 0x7fae08155910 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fadf00011f0 tx=0x7fadf000c9b0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 shutdown_connections 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fade8066470 0x7fade8068930 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fae0815d2b0 0x7fae0814e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae080d1cf0 0x7fae08155910 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 --2- 172.21.15.28:0/1680461538 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae080d02f0 0x7fae0815a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 >> 172.21.15.28:0/1680461538 conn(0x7fae0809c510 msgr2=0x7fae080b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 shutdown_connections 2023-12-16T21:58:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.230+0000 7fae0ceff640 1 -- 172.21.15.28:0/1680461538 wait complete. 2023-12-16T21:58:23.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:23.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb821f9d640 1 Processor -- start 2023-12-16T21:58:23.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb821f9d640 1 -- start start 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb821f9d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c15eb50 0x7fb81c06f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb821f9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb81c0703a0 0x7fb81c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb821f9d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c06e5c0 con 0x7fb81c0cf4d0 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb821f9d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb81c09a7c0 con 0x7fb81c0703a0 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb821f9d640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c06e810 con 0x7fb81c15eb50 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81b7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb81c0cf4d0 0x7fb81c15e440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47496/0 (socket says 172.21.15.28:47496) 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81b7fe640 1 -- 172.21.15.28:0/781487184 learned_addr learned my addr 172.21.15.28:0/781487184 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81bfff640 1 --2- 172.21.15.28:0/781487184 >> v2:172.21.15.28:3300/0 conn(0x7fb81c0703a0 0x7fb81c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81affd640 1 --2- 172.21.15.28:0/781487184 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c15eb50 0x7fb81c06f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81a7fc640 1 -- 172.21.15.28:0/781487184 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872303601 0 0) 0x7fb81c06e5c0 con 0x7fb81c0cf4d0 2023-12-16T21:58:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81a7fc640 1 -- 172.21.15.28:0/781487184 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb808003530 con 0x7fb81c0cf4d0 2023-12-16T21:58:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81bfff640 1 -- 172.21.15.28:0/781487184 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c15eb50 msgr2=0x7fb81c06f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81bfff640 1 --2- 172.21.15.28:0/781487184 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c15eb50 0x7fb81c06f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81bfff640 1 -- 172.21.15.28:0/781487184 >> v1:172.21.15.28:6789/0 conn(0x7fb81c0cf4d0 legacy=0x7fb81c15e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.370+0000 7fb81bfff640 1 -- 172.21.15.28:0/781487184 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c06eaf0 con 0x7fb81c0703a0 2023-12-16T21:58:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb81bfff640 1 --2- 172.21.15.28:0/781487184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0cb9e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb814009d90 tx=0x7fb81401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05cbff2510c0a3a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb81a7fc640 1 -- 172.21.15.28:0/781487184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8140a5a00 con 0x7fb81c0703a0 2023-12-16T21:58:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb81a7fc640 1 -- 172.21.15.28:0/781487184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8140a5b60 con 0x7fb81c0703a0 2023-12-16T21:58:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb81a7fc640 1 -- 172.21.15.28:0/781487184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8140ad440 con 0x7fb81c0703a0 2023-12-16T21:58:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 -- 172.21.15.28:0/781487184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 msgr2=0x7fb81c0cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 --2- 172.21.15.28:0/781487184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0cb9e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb814009d90 tx=0x7fb81401c920 comp rx=0 tx=0).stop 2023-12-16T21:58:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 -- 172.21.15.28:0/781487184 shutdown_connections 2023-12-16T21:58:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 --2- 172.21.15.28:0/781487184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0cb9e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 --2- 172.21.15.28:0/781487184 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c15eb50 0x7fb81c06f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 -- 172.21.15.28:0/781487184 >> 172.21.15.28:0/781487184 conn(0x7fb81c0b2aa0 msgr2=0x7fb81c0b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 -- 172.21.15.28:0/781487184 shutdown_connections 2023-12-16T21:58:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 -- 172.21.15.28:0/781487184 wait complete. 2023-12-16T21:58:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 Processor -- start 2023-12-16T21:58:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 -- start start 2023-12-16T21:58:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c0cf4d0 0x7fb81c0c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.374+0000 7fb821f9d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb81c15eb50 0x7fb81c0c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb821f9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c0b67a0 con 0x7fb81c0703a0 2023-12-16T21:58:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb821f9d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c15cbd0 con 0x7fb81c15eb50 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb821f9d640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c15ceb0 con 0x7fb81c0cf4d0 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0b6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46686/0 (socket says 172.21.15.28:46686) 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 -- 172.21.15.28:0/2032375465 learned_addr learned my addr 172.21.15.28:0/2032375465 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81affd640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c0cf4d0 0x7fb81c0c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81bfff640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb81c15eb50 0x7fb81c0c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 -- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c0cf4d0 msgr2=0x7fb81c0c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c0cf4d0 0x7fb81c0c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 -- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb81c15eb50 msgr2=0x7fb81c0c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb81c15eb50 0x7fb81c0c85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 -- 172.21.15.28:0/2032375465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb814009a70 con 0x7fb81c0703a0 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81affd640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c0cf4d0 0x7fb81c0c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81b7fe640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0b6260 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb80400c450 tx=0x7fb80400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb813fff640 1 -- 172.21.15.28:0/2032375465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb804005cf0 con 0x7fb81c0703a0 2023-12-16T21:58:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb813fff640 1 -- 172.21.15.28:0/2032375465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb804005e50 con 0x7fb81c0703a0 2023-12-16T21:58:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c15d190 con 0x7fb81c0703a0 2023-12-16T21:58:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb813fff640 1 -- 172.21.15.28:0/2032375465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb804007290 con 0x7fb81c0703a0 2023-12-16T21:58:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c15be30 con 0x7fb81c0703a0 2023-12-16T21:58:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c067210 con 0x7fb81c0703a0 2023-12-16T21:58:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb813fff640 1 -- 172.21.15.28:0/2032375465 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fb8040073f0 con 0x7fb81c0703a0 2023-12-16T21:58:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb813fff640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb7fc0664a0 0x7fb7fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb813fff640 1 -- 172.21.15.28:0/2032375465 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb804083790 con 0x7fb81c0703a0 2023-12-16T21:58:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.378+0000 7fb81affd640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb7fc0664a0 0x7fb7fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.382+0000 7fb81affd640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb7fc0664a0 0x7fb7fc068960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb81c15dd00 tx=0x7fb80c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.382+0000 7fb813fff640 1 -- 172.21.15.28:0/2032375465 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fb804050d80 con 0x7fb81c0703a0 2023-12-16T21:58:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.542+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb81c0b6be0 con 0x7fb81c0703a0 2023-12-16T21:58:23.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.542+0000 7fb813fff640 1 -- 172.21.15.28:0/2032375465 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb8040543a0 con 0x7fb81c0703a0 2023-12-16T21:58:23.545 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb7fc0664a0 msgr2=0x7fb7fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb7fc0664a0 0x7fb7fc068960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb81c15dd00 tx=0x7fb80c008040 comp rx=0 tx=0).stop 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 msgr2=0x7fb81c0b6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0b6260 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb80400c450 tx=0x7fb80400c910 comp rx=0 tx=0).stop 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 shutdown_connections 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb7fc0664a0 0x7fb7fc068960 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb81c15eb50 0x7fb81c0c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb81c0cf4d0 0x7fb81c0c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 --2- 172.21.15.28:0/2032375465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb81c0703a0 0x7fb81c0b6260 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.548 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 >> 172.21.15.28:0/2032375465 conn(0x7fb81c0b2aa0 msgr2=0x7fb81c0b2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 shutdown_connections 2023-12-16T21:58:23.549 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.546+0000 7fb821f9d640 1 -- 172.21.15.28:0/2032375465 wait complete. 2023-12-16T21:58:23.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:23.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aab003640 1 Processor -- start 2023-12-16T21:58:23.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aab003640 1 -- start start 2023-12-16T21:58:23.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aab003640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1aa40ae830 0x7f1aa415d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aab003640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 0x7f1aa40ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aab003640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa41575b0 con 0x7f1aa41571d0 2023-12-16T21:58:23.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aab003640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa415db80 con 0x7f1aa40ae830 2023-12-16T21:58:23.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aab003640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa4071390 con 0x7f1aa415d710 2023-12-16T21:58:23.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa8d9f640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1aa41571d0 0x7f1aa4070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47500/0 (socket says 172.21.15.28:47500) 2023-12-16T21:58:23.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa8d9f640 1 -- 172.21.15.28:0/3458507422 learned_addr learned my addr 172.21.15.28:0/3458507422 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:23.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992669601 0 0) 0x7f1aa41575b0 con 0x7f1aa41571d0 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa3fff640 1 --2- 172.21.15.28:0/3458507422 >> v2:172.21.15.122:3300/0 conn(0x7f1aa40ae830 0x7f1aa415d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa95a0640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 0x7f1aa40ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a8c003530 con 0x7f1aa41571d0 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2040609898 0 0) 0x7f1a8c003530 con 0x7f1aa41571d0 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 msgr2=0x7f1aa40ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 0x7f1aa40ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 >> v2:172.21.15.122:3300/0 conn(0x7f1aa40ae830 msgr2=0x7f1aa415d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 --2- 172.21.15.28:0/3458507422 >> v2:172.21.15.122:3300/0 conn(0x7f1aa40ae830 0x7f1aa415d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa4071600 con 0x7f1aa41571d0 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f1a94001680 con 0x7f1aa41571d0 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa3fff640 1 --2- 172.21.15.28:0/3458507422 >> v2:172.21.15.122:3300/0 conn(0x7f1aa40ae830 0x7f1aa415d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa95a0640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 0x7f1aa40ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 >> v1:172.21.15.28:6789/0 conn(0x7f1aa41571d0 legacy=0x7f1aa4070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c003980 0x7f1a8c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c0047d0 0x7f1a8c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 0x7f1a8c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa415db80 con 0x7f1a8c0047d0 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa4071390 con 0x7f1a8c003980 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c00ba10 con 0x7f1aa415d710 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa3fff640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c0047d0 0x7f1a8c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa8d9f640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c003980 0x7f1a8c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa95a0640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 0x7f1a8c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa3fff640 1 -- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 msgr2=0x7f1a8c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa3fff640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 0x7f1a8c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa3fff640 1 -- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c003980 msgr2=0x7f1a8c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa3fff640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1a8c003980 0x7f1a8c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa3fff640 1 -- 172.21.15.28:0/3458507422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c003530 con 0x7f1a8c0047d0 2023-12-16T21:58:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.690+0000 7f1aa95a0640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1aa415d710 0x7f1a8c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aa3fff640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c0047d0 0x7f1a8c004c50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1a980098d0 tx=0x7f1a9801c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86c52f1ddc4ec61 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980aa5f0 con 0x7f1a8c0047d0 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a980aa750 con 0x7f1a8c0047d0 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aa37fe640 1 -- 172.21.15.28:0/3458507422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a980aaac0 con 0x7f1a8c0047d0 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aab003640 1 -- 172.21.15.28:0/3458507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c0047d0 msgr2=0x7f1a8c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aab003640 1 --2- 172.21.15.28:0/3458507422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c0047d0 0x7f1a8c004c50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1a980098d0 tx=0x7f1a9801c630 comp rx=0 tx=0).stop 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aa95a0640 1 -- 172.21.15.28:0/3458507422 reap_dead start 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aab003640 1 -- 172.21.15.28:0/3458507422 shutdown_connections 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aab003640 1 -- 172.21.15.28:0/3458507422 >> 172.21.15.28:0/3458507422 conn(0x7f1aa40b0cc0 msgr2=0x7f1aa40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aab003640 1 -- 172.21.15.28:0/3458507422 shutdown_connections 2023-12-16T21:58:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aab003640 1 -- 172.21.15.28:0/3458507422 wait complete. 2023-12-16T21:58:23.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aab003640 1 Processor -- start 2023-12-16T21:58:23.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.694+0000 7f1aab003640 1 -- start start 2023-12-16T21:58:23.701 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1a8c0047d0 0x7f1aa4136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.701 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa414c250 0x7f1aa40b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.701 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa40b2c80 0x7f1aa4152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa40ad560 con 0x7f1aa40b2c80 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa414caf0 con 0x7f1aa414c250 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa413a770 con 0x7f1a8c0047d0 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa40b2c80 0x7f1aa4152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa40b2c80 0x7f1aa4152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46702/0 (socket says 172.21.15.28:46702) 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 -- 172.21.15.28:0/3489631930 learned_addr learned my addr 172.21.15.28:0/3489631930 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa8d9f640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1a8c0047d0 0x7f1aa4136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa3fff640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa414c250 0x7f1aa40b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 -- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1a8c0047d0 msgr2=0x7f1aa4136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1a8c0047d0 0x7f1aa4136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 -- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa414c250 msgr2=0x7f1aa40b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa414c250 0x7f1aa40b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 -- 172.21.15.28:0/3489631930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa41354f0 con 0x7f1aa40b2c80 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa95a0640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa40b2c80 0x7f1aa4152ca0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40703a0 tx=0x7f1a9c004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa17fa640 1 -- 172.21.15.28:0/3489631930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c007c70 con 0x7f1aa40b2c80 2023-12-16T21:58:23.703 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa17fa640 1 -- 172.21.15.28:0/3489631930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c007e10 con 0x7f1aa40b2c80 2023-12-16T21:58:23.703 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa40b4280 con 0x7f1aa40b2c80 2023-12-16T21:58:23.703 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa17fa640 1 -- 172.21.15.28:0/3489631930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c014640 con 0x7f1aa40b2c80 2023-12-16T21:58:23.703 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa4135790 con 0x7f1aa40b2c80 2023-12-16T21:58:23.703 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa4135980 con 0x7f1aa40b2c80 2023-12-16T21:58:23.704 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa17fa640 1 -- 172.21.15.28:0/3489631930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1a9c0147a0 con 0x7f1aa40b2c80 2023-12-16T21:58:23.704 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa17fa640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a90066470 0x7f1a90068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.704 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa17fa640 1 -- 172.21.15.28:0/3489631930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a9c083980 con 0x7f1aa40b2c80 2023-12-16T21:58:23.704 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.698+0000 7f1aa8d9f640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a90066470 0x7f1a90068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.704 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.702+0000 7f1aa8d9f640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a90066470 0x7f1a90068930 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1a94001630 tx=0x7f1a940015c0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:23.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.702+0000 7f1aa17fa640 1 -- 172.21.15.28:0/3489631930 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f1a9c050f40 con 0x7f1aa40b2c80 2023-12-16T21:58:23.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.858+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1aa40693d0 con 0x7f1aa40b2c80 2023-12-16T21:58:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.858+0000 7f1aa17fa640 1 -- 172.21.15.28:0/3489631930 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a9c019020 con 0x7f1aa40b2c80 2023-12-16T21:58:23.862 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a90066470 msgr2=0x7f1a90068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a90066470 0x7f1a90068930 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1a94001630 tx=0x7f1a940015c0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa40b2c80 msgr2=0x7f1aa4152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa40b2c80 0x7f1aa4152ca0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40703a0 tx=0x7f1a9c004700 comp rx=0 tx=0).stop 2023-12-16T21:58:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 shutdown_connections 2023-12-16T21:58:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1a90066470 0x7f1a90068930 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa40b2c80 0x7f1aa4152ca0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1aa414c250 0x7f1aa40b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 --2- 172.21.15.28:0/3489631930 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1a8c0047d0 0x7f1aa4136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 >> 172.21.15.28:0/3489631930 conn(0x7f1aa40b0cc0 msgr2=0x7f1aa40af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:23.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 shutdown_connections 2023-12-16T21:58:23.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.862+0000 7f1aab003640 1 -- 172.21.15.28:0/3489631930 wait complete. 2023-12-16T21:58:23.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:58:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29be7fc640 1 Processor -- start 2023-12-16T21:58:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29be7fc640 1 -- start start 2023-12-16T21:58:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29be7fc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f29b80af8b0 0x7f29b8167fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29be7fc640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80adfb0 con 0x7f29b8071fd0 2023-12-16T21:58:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29be7fc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80ae290 con 0x7f29b815d3b0 2023-12-16T21:58:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29be7fc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b80aeb30 con 0x7f29b80af8b0 2023-12-16T21:58:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b77fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29b8071fd0 0x7f29b80af1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47514/0 (socket says 172.21.15.28:47514) 2023-12-16T21:58:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b77fe640 1 -- 172.21.15.28:0/4182200302 learned_addr learned my addr 172.21.15.28:0/4182200302 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29bcd99640 1 --2- 172.21.15.28:0/4182200302 >> v2:172.21.15.122:3300/0 conn(0x7f29b80af8b0 0x7f29b8167fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684075118 0 0) 0x7f29b80adfb0 con 0x7f29b8071fd0 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a0003530 con 0x7f29b8071fd0 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1344872075 0 0) 0x7f29a0003530 con 0x7f29b8071fd0 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 >> v2:172.21.15.122:3300/0 conn(0x7f29b80af8b0 msgr2=0x7f29b8167fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 --2- 172.21.15.28:0/4182200302 >> v2:172.21.15.122:3300/0 conn(0x7f29b80af8b0 0x7f29b8167fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 >> v1:172.21.15.122:6789/0 conn(0x7f29b815d3b0 legacy=0x7f29b80ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b80ae570 con 0x7f29b8071fd0 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f29ac001680 con 0x7f29b8071fd0 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 >> v1:172.21.15.28:6789/0 conn(0x7f29b8071fd0 legacy=0x7f29b80af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0003980 0x7f29a0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.996 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a00047d0 0x7f29a0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a0005190 0x7f29a000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b80aeb30 con 0x7f29a0005190 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a000c040 con 0x7f29a00047d0 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a000c200 con 0x7f29a0003980 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29bcd99640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a0005190 0x7f29a000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b77fe640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a00047d0 0x7f29a0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29b7fff640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0003980 0x7f29a0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29bcd99640 1 -- 172.21.15.28:0/4182200302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0003980 msgr2=0x7f29a0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29bcd99640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0003980 0x7f29a0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29bcd99640 1 -- 172.21.15.28:0/4182200302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a00047d0 msgr2=0x7f29a0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29bcd99640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29a00047d0 0x7f29a0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.990+0000 7f29bcd99640 1 -- 172.21.15.28:0/4182200302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a0003530 con 0x7f29a0005190 2023-12-16T21:58:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29bcd99640 1 -- 172.21.15.28:0/4182200302 reap_dead start 2023-12-16T21:58:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29bcd99640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a0005190 0x7f29a000b990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009fc0 tx=0x7f29b001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff7b3c65e9aea19 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.001 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b00a5cb0 con 0x7f29a0005190 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b000f040 con 0x7f29a0005190 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29b6ffd640 1 -- 172.21.15.28:0/4182200302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0003be0 con 0x7f29a0005190 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 -- 172.21.15.28:0/4182200302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a0005190 msgr2=0x7f29a000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a0005190 0x7f29a000b990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009fc0 tx=0x7f29b001cca0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 -- 172.21.15.28:0/4182200302 shutdown_connections 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 --2- 172.21.15.28:0/4182200302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a0005190 0x7f29a000b990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 -- 172.21.15.28:0/4182200302 >> 172.21.15.28:0/4182200302 conn(0x7f29b806b640 msgr2=0x7f29b806ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 -- 172.21.15.28:0/4182200302 shutdown_connections 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 -- 172.21.15.28:0/4182200302 wait complete. 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 Processor -- start 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.994+0000 7f29be7fc640 1 -- start start 2023-12-16T21:58:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0005190 0x7f29b80b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29b815f3b0 0x7f29b815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b80c56f0 0x7f29b80c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b80c52d0 con 0x7f29b80c56f0 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b8160050 con 0x7f29b815f3b0 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b8160330 con 0x7f29a0005190 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b80c56f0 0x7f29b80c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b80c56f0 0x7f29b80c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46728/0 (socket says 172.21.15.28:46728) 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 -- 172.21.15.28:0/3552912867 learned_addr learned my addr 172.21.15.28:0/3552912867 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29b7fff640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0005190 0x7f29b80b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 -- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0005190 msgr2=0x7f29b80b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0005190 0x7f29b80b3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 -- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29b815f3b0 msgr2=0x7f29b815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29b815f3b0 0x7f29b815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 -- 172.21.15.28:0/3552912867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b8160610 con 0x7f29b80c56f0 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29bcd99640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b80c56f0 0x7f29b80c5b70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f29b00987d0 tx=0x7f29b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29b4ff9640 1 -- 172.21.15.28:0/3552912867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b00aed40 con 0x7f29b80c56f0 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29b4ff9640 1 -- 172.21.15.28:0/3552912867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b00aeec0 con 0x7f29b80c56f0 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b815fd50 con 0x7f29b80c56f0 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29b4ff9640 1 -- 172.21.15.28:0/3552912867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0003be0 con 0x7f29b80c56f0 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b80b46d0 con 0x7f29b80c56f0 2023-12-16T21:58:24.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b8160850 con 0x7f29b80c56f0 2023-12-16T21:58:24.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29b4ff9640 1 -- 172.21.15.28:0/3552912867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f29b0003d40 con 0x7f29b80c56f0 2023-12-16T21:58:24.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29b4ff9640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f29a4066470 0x7f29a4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29b4ff9640 1 -- 172.21.15.28:0/3552912867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f29b00a6e40 con 0x7f29b80c56f0 2023-12-16T21:58:24.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:23.998+0000 7f29b7fff640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f29a4066470 0x7f29a4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.002+0000 7f29b7fff640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f29a4066470 0x7f29a4068930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f29a80015a0 tx=0x7f29a8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.002+0000 7f29b4ff9640 1 -- 172.21.15.28:0/3552912867 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f29b8160850 con 0x7f29b80c56f0 2023-12-16T21:58:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.170+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f29b80c5410 con 0x7f29b80c56f0 2023-12-16T21:58:24.174 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.170+0000 7f29b4ff9640 1 -- 172.21.15.28:0/3552912867 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f29b80c5410 con 0x7f29b80c56f0 2023-12-16T21:58:24.174 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2023-12-16T21:58:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f29a4066470 msgr2=0x7f29a4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f29a4066470 0x7f29a4068930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f29a80015a0 tx=0x7f29a8009040 comp rx=0 tx=0).stop 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b80c56f0 msgr2=0x7f29b80c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b80c56f0 0x7f29b80c5b70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f29b00987d0 tx=0x7f29b0009750 comp rx=0 tx=0).stop 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 shutdown_connections 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f29a4066470 0x7f29a4068930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b80c56f0 0x7f29b80c5b70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f29b815f3b0 0x7f29b815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 --2- 172.21.15.28:0/3552912867 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f29a0005190 0x7f29b80b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 >> 172.21.15.28:0/3552912867 conn(0x7f29b806b640 msgr2=0x7f29b815c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 shutdown_connections 2023-12-16T21:58:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.174+0000 7f29be7fc640 1 -- 172.21.15.28:0/3552912867 wait complete. 2023-12-16T21:58:24.194 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2023-12-16T21:58:24.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:24.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-12-16T21:58:24.206 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:58:24.206 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2023-12-16T21:58:24.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:24.207 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-16T21:58:24.252 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:58:24.252 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:24.259 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:58:24.259 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:24.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:24.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.426+0000 7f69f89e4640 1 Processor -- start 2023-12-16T21:58:24.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.426+0000 7f69f89e4640 1 -- start start 2023-12-16T21:58:24.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f89e4640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69f40b3bc0 0x7f69f40b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f89e4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f69f40b2f40 0x7f69f40d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f89e4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f40cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f89e4640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f40d2150 con 0x7f69f40b3bc0 2023-12-16T21:58:24.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f89e4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f4160f80 con 0x7f69f40b2f40 2023-12-16T21:58:24.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f89e4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40d0d30 con 0x7f69f40d1cc0 2023-12-16T21:58:24.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69f40b3bc0 0x7f69f40b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69f40b3bc0 0x7f69f40b2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46744/0 (socket says 172.21.15.28:46744) 2023-12-16T21:58:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 -- 172.21.15.28:0/4192586277 learned_addr learned my addr 172.21.15.28:0/4192586277 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f1d9b640 1 --2- 172.21.15.28:0/4192586277 >> v2:172.21.15.122:3300/0 conn(0x7f69f40b2f40 0x7f69f40d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f2d9d640 1 --2- 172.21.15.28:0/4192586277 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f40cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 -- 172.21.15.28:0/4192586277 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 msgr2=0x7f69f40cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 --2- 172.21.15.28:0/4192586277 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f40cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 -- 172.21.15.28:0/4192586277 >> v2:172.21.15.122:3300/0 conn(0x7f69f40b2f40 msgr2=0x7f69f40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 --2- 172.21.15.28:0/4192586277 >> v2:172.21.15.122:3300/0 conn(0x7f69f40b2f40 0x7f69f40d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 -- 172.21.15.28:0/4192586277 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f40d12f0 con 0x7f69f40b3bc0 2023-12-16T21:58:24.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.430+0000 7f69f259c640 1 --2- 172.21.15.28:0/4192586277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b3bc0 0x7f69f40b2a00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f69f4073f00 tx=0x7f69ec01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2f753344b1a956 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f159a640 1 -- 172.21.15.28:0/4192586277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0ac600 con 0x7f69f40b3bc0 2023-12-16T21:58:24.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f159a640 1 -- 172.21.15.28:0/4192586277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec0a8050 con 0x7f69f40b3bc0 2023-12-16T21:58:24.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f159a640 1 -- 172.21.15.28:0/4192586277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0ac980 con 0x7f69f40b3bc0 2023-12-16T21:58:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- 172.21.15.28:0/4192586277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b3bc0 msgr2=0x7f69f40b2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 --2- 172.21.15.28:0/4192586277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b3bc0 0x7f69f40b2a00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f69f4073f00 tx=0x7f69ec01c4d0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- 172.21.15.28:0/4192586277 shutdown_connections 2023-12-16T21:58:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 --2- 172.21.15.28:0/4192586277 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f40cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 --2- 172.21.15.28:0/4192586277 >> v2:172.21.15.122:3300/0 conn(0x7f69f40b2f40 0x7f69f40d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 --2- 172.21.15.28:0/4192586277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b3bc0 0x7f69f40b2a00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- 172.21.15.28:0/4192586277 >> 172.21.15.28:0/4192586277 conn(0x7f69f409c430 msgr2=0x7f69f40d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:24.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- 172.21.15.28:0/4192586277 shutdown_connections 2023-12-16T21:58:24.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- 172.21.15.28:0/4192586277 wait complete. 2023-12-16T21:58:24.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 Processor -- start 2023-12-16T21:58:24.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- start start 2023-12-16T21:58:24.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b2f40 0x7f69f4144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40b3bc0 0x7f69f4157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f4153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f41560c0 con 0x7f69f40b2f40 2023-12-16T21:58:24.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f41563a0 con 0x7f69f40b3bc0 2023-12-16T21:58:24.439 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4156680 con 0x7f69f40d1cc0 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b2f40 0x7f69f4144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b2f40 0x7f69f4144790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46760/0 (socket says 172.21.15.28:46760) 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 -- 172.21.15.28:0/2950261508 learned_addr learned my addr 172.21.15.28:0/2950261508 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f1d9b640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40b3bc0 0x7f69f4157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f2d9d640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f4153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 -- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 msgr2=0x7f69f4153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f4153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 -- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40b3bc0 msgr2=0x7f69f4157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40b3bc0 0x7f69f4157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 -- 172.21.15.28:0/2950261508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec0094d0 con 0x7f69f40b2f40 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f2d9d640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f4153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f259c640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b2f40 0x7f69f4144790 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0097c0 tx=0x7f69ec019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69eaffd640 1 -- 172.21.15.28:0/2950261508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0b5610 con 0x7f69f40b2f40 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69eaffd640 1 -- 172.21.15.28:0/2950261508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec0b5770 con 0x7f69f40b2f40 2023-12-16T21:58:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.434+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f4156960 con 0x7f69f40b2f40 2023-12-16T21:58:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.438+0000 7f69eaffd640 1 -- 172.21.15.28:0/2950261508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0b5b20 con 0x7f69f40b2f40 2023-12-16T21:58:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.438+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f4144cd0 con 0x7f69f40b2f40 2023-12-16T21:58:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.438+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f40674d0 con 0x7f69f40b2f40 2023-12-16T21:58:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.438+0000 7f69eaffd640 1 -- 172.21.15.28:0/2950261508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f69ec0b3040 con 0x7f69f40b2f40 2023-12-16T21:58:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.438+0000 7f69eaffd640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69d8066470 0x7f69d8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.438+0000 7f69eaffd640 1 -- 172.21.15.28:0/2950261508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f69ec11de20 con 0x7f69f40b2f40 2023-12-16T21:58:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.438+0000 7f69f1d9b640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69d8066470 0x7f69d8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.438+0000 7f69f1d9b640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69d8066470 0x7f69d8068930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f69f40b3340 tx=0x7f69dc009790 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.445 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.442+0000 7f69eaffd640 1 -- 172.21.15.28:0/2950261508 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f69ec0eb350 con 0x7f69f40b2f40 2023-12-16T21:58:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.594+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69f40d07e0 con 0x7f69f40b2f40 2023-12-16T21:58:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.598+0000 7f69eaffd640 1 -- 172.21.15.28:0/2950261508 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f69ec0ee970 con 0x7f69f40b2f40 2023-12-16T21:58:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:24.601 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:23.748909+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.598+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69d8066470 msgr2=0x7f69d8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.598+0000 7f69f89e4640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69d8066470 0x7f69d8068930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f69f40b3340 tx=0x7f69dc009790 comp rx=0 tx=0).stop 2023-12-16T21:58:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.598+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b2f40 msgr2=0x7f69f4144790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.598+0000 7f69f89e4640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b2f40 0x7f69f4144790 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0097c0 tx=0x7f69ec019270 comp rx=0 tx=0).stop 2023-12-16T21:58:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.602+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 shutdown_connections 2023-12-16T21:58:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.602+0000 7f69f89e4640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69d8066470 0x7f69d8068930 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.602+0000 7f69f89e4640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69f40d1cc0 0x7f69f4153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.602+0000 7f69f89e4640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69f40b3bc0 0x7f69f4157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.602+0000 7f69f89e4640 1 --2- 172.21.15.28:0/2950261508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69f40b2f40 0x7f69f4144790 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.602+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 >> 172.21.15.28:0/2950261508 conn(0x7f69f409c430 msgr2=0x7f69f406caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.602+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 shutdown_connections 2023-12-16T21:58:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.602+0000 7f69f89e4640 1 -- 172.21.15.28:0/2950261508 wait complete. 2023-12-16T21:58:24.619 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:24.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:24.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f011c04b640 1 Processor -- start 2023-12-16T21:58:24.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f011c04b640 1 -- start start 2023-12-16T21:58:24.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f011c04b640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0114158ca0 0x7f011415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f011c04b640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011415b670 con 0x7f01140cba10 2023-12-16T21:58:24.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f011c04b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011415c510 con 0x7f01140b42d0 2023-12-16T21:58:24.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f011c04b640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011415f410 con 0x7f0114158ca0 2023-12-16T21:58:24.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f01195e6640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f01140cba10 0x7f011415be00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47520/0 (socket says 172.21.15.28:47520) 2023-12-16T21:58:24.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f01195e6640 1 -- 172.21.15.28:0/1840752797 learned_addr learned my addr 172.21.15.28:0/1840752797 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0119de7640 1 --2- 172.21.15.28:0/1840752797 >> v2:172.21.15.122:3300/0 conn(0x7f0114158ca0 0x7f011415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801416767 0 0) 0x7f011415b670 con 0x7f01140cba10 2023-12-16T21:58:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0100003530 con 0x7f01140cba10 2023-12-16T21:58:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811565158 0 0) 0x7f0100003530 con 0x7f01140cba10 2023-12-16T21:58:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 >> v2:172.21.15.122:3300/0 conn(0x7f0114158ca0 msgr2=0x7f011415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 --2- 172.21.15.28:0/1840752797 >> v2:172.21.15.122:3300/0 conn(0x7f0114158ca0 0x7f011415eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 >> v1:172.21.15.122:6789/0 conn(0x7f01140b42d0 legacy=0x7f01140703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011415f6f0 con 0x7f01140cba10 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f0110001680 con 0x7f01140cba10 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 >> v1:172.21.15.28:6789/0 conn(0x7f01140cba10 legacy=0x7f011415be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0100003980 0x7f0100003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f0100004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0100005190 0x7f010000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011415f410 con 0x7f01000047d0 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010000c0e0 con 0x7f0100005190 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010000c2a0 con 0x7f0100003980 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.746+0000 7f01195e6640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f0100004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f0119de7640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0100003980 0x7f0100003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011a5e8640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0100005190 0x7f010000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f01195e6640 1 -- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0100003980 msgr2=0x7f0100003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f01195e6640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0100003980 0x7f0100003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f01195e6640 1 -- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0100005190 msgr2=0x7f010000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f01195e6640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0100005190 0x7f010000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f01195e6640 1 -- 172.21.15.28:0/1840752797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100003530 con 0x7f01000047d0 2023-12-16T21:58:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011a5e8640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0100005190 0x7f010000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011a5e8640 1 -- 172.21.15.28:0/1840752797 reap_dead start 2023-12-16T21:58:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f0119de7640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0100003980 0x7f0100003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f01195e6640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f0100004c50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f011000c210 tx=0x7f011001f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b1f7afd932b8d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110005ae0 con 0x7f01000047d0 2023-12-16T21:58:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01100ab070 con 0x7f01000047d0 2023-12-16T21:58:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f0118de5640 1 -- 172.21.15.28:0/1840752797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110007070 con 0x7f01000047d0 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011c04b640 1 -- 172.21.15.28:0/1840752797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 msgr2=0x7f0100004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011c04b640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f0100004c50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f011000c210 tx=0x7f011001f370 comp rx=0 tx=0).stop 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011c04b640 1 -- 172.21.15.28:0/1840752797 shutdown_connections 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011c04b640 1 --2- 172.21.15.28:0/1840752797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f0100004c50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011c04b640 1 -- 172.21.15.28:0/1840752797 >> 172.21.15.28:0/1840752797 conn(0x7f01140b0cc0 msgr2=0x7f01140b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011c04b640 1 -- 172.21.15.28:0/1840752797 shutdown_connections 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.750+0000 7f011c04b640 1 -- 172.21.15.28:0/1840752797 wait complete. 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 Processor -- start 2023-12-16T21:58:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 -- start start 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f01141400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f011413d030 0x7f0114153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0114153b50 0x7f011415d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0114153fd0 con 0x7f01000047d0 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0114156e10 con 0x7f011413d030 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0114140600 con 0x7f0114153b50 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f01141400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f01141400c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46792/0 (socket says 172.21.15.28:46792) 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 -- 172.21.15.28:0/1625122135 learned_addr learned my addr 172.21.15.28:0/1625122135 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011a5e8640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0114153b50 0x7f011415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f01195e6640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f011413d030 0x7f0114153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 -- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0114153b50 msgr2=0x7f011415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0114153b50 0x7f011415d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 -- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f011413d030 msgr2=0x7f0114153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f011413d030 0x7f0114153610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 -- 172.21.15.28:0/1625122135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01141e4bd0 con 0x7f01000047d0 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f0119de7640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f01141400c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f010400e570 tx=0x7f010400ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f010a7fc640 1 -- 172.21.15.28:0/1625122135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0104013610 con 0x7f01000047d0 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f010a7fc640 1 -- 172.21.15.28:0/1625122135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0104013770 con 0x7f01000047d0 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01141567c0 con 0x7f01000047d0 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f010a7fc640 1 -- 172.21.15.28:0/1625122135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0104013b00 con 0x7f01000047d0 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0114156470 con 0x7f01000047d0 2023-12-16T21:58:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01140693d0 con 0x7f01000047d0 2023-12-16T21:58:24.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f010a7fc640 1 -- 172.21.15.28:0/1625122135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f0104013c60 con 0x7f01000047d0 2023-12-16T21:58:24.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f010a7fc640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00f8066470 0x7f00f8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:24.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.754+0000 7f010a7fc640 1 -- 172.21.15.28:0/1625122135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0104083bf0 con 0x7f01000047d0 2023-12-16T21:58:24.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.758+0000 7f01195e6640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00f8066470 0x7f00f8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:24.761 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.758+0000 7f01195e6640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00f8066470 0x7f00f8068930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f01100af000 tx=0x7f01100aa000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:24.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.762+0000 7f010a7fc640 1 -- 172.21.15.28:0/1625122135 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f0104050fd0 con 0x7f01000047d0 2023-12-16T21:58:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f01141e4f90 con 0x7f01000047d0 2023-12-16T21:58:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f010a7fc640 1 -- 172.21.15.28:0/1625122135 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f01040210a0 con 0x7f01000047d0 2023-12-16T21:58:24.921 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00f8066470 msgr2=0x7f00f8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00f8066470 0x7f00f8068930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f01100af000 tx=0x7f01100aa000 comp rx=0 tx=0).stop 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 msgr2=0x7f01141400c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f01141400c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f010400e570 tx=0x7f010400ea30 comp rx=0 tx=0).stop 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 shutdown_connections 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00f8066470 0x7f00f8068930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0114153b50 0x7f011415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f011413d030 0x7f0114153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 --2- 172.21.15.28:0/1625122135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01000047d0 0x7f01141400c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.918+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 >> 172.21.15.28:0/1625122135 conn(0x7f01140b0cc0 msgr2=0x7f01140cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.922+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 shutdown_connections 2023-12-16T21:58:24.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:24.922+0000 7f011c04b640 1 -- 172.21.15.28:0/1625122135 wait complete. 2023-12-16T21:58:24.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:25.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.054+0000 7f6850cab640 1 Processor -- start 2023-12-16T21:58:25.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.054+0000 7f6850cab640 1 -- start start 2023-12-16T21:58:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f6850cab640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f6850cab640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f684c13f5f0 0x7f684c0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f6850cab640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684c13d2b0 con 0x7f684c15adb0 2023-12-16T21:58:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f6850cab640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684c140520 con 0x7f684c13f5f0 2023-12-16T21:58:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f6850cab640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0cd7f0 con 0x7f684c071d90 2023-12-16T21:58:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f684a59c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f684ad9d640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f684c13f5f0 0x7f684c0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f6849d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f684c15adb0 0x7f684c13d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:53082/0 (socket says 172.21.15.28:53082) 2023-12-16T21:58:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f6849d9b640 1 -- 172.21.15.28:0/258120665 learned_addr learned my addr 172.21.15.28:0/258120665 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f684a59c640 1 --2- 172.21.15.28:0/258120665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c072170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:49454/0 (socket says 172.21.15.28:49454) 2023-12-16T21:58:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f684a59c640 1 -- 172.21.15.28:0/258120665 >> v2:172.21.15.122:3300/0 conn(0x7f684c13f5f0 msgr2=0x7f684c0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f684a59c640 1 --2- 172.21.15.28:0/258120665 >> v2:172.21.15.122:3300/0 conn(0x7f684c13f5f0 0x7f684c0ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f684a59c640 1 -- 172.21.15.28:0/258120665 >> v1:172.21.15.122:6789/0 conn(0x7f684c15adb0 legacy=0x7f684c13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.058+0000 7f684a59c640 1 -- 172.21.15.28:0/258120665 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c0cdd70 con 0x7f684c071d90 2023-12-16T21:58:25.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684a59c640 1 --2- 172.21.15.28:0/258120665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c072170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f683c00ba60 tx=0x7f683c01eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc1d29cdecd871f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.063 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684959a640 1 -- 172.21.15.28:0/258120665 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0a7b20 con 0x7f684c071d90 2023-12-16T21:58:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684959a640 1 -- 172.21.15.28:0/258120665 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c0a7c80 con 0x7f684c071d90 2023-12-16T21:58:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684959a640 1 -- 172.21.15.28:0/258120665 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c0af440 con 0x7f684c071d90 2023-12-16T21:58:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- 172.21.15.28:0/258120665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 msgr2=0x7f684c072170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 --2- 172.21.15.28:0/258120665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c072170 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f683c00ba60 tx=0x7f683c01eb50 comp rx=0 tx=0).stop 2023-12-16T21:58:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- 172.21.15.28:0/258120665 shutdown_connections 2023-12-16T21:58:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 --2- 172.21.15.28:0/258120665 >> v2:172.21.15.122:3300/0 conn(0x7f684c13f5f0 0x7f684c0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 --2- 172.21.15.28:0/258120665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c072170 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- 172.21.15.28:0/258120665 >> 172.21.15.28:0/258120665 conn(0x7f684c0b0cc0 msgr2=0x7f684c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- 172.21.15.28:0/258120665 shutdown_connections 2023-12-16T21:58:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- 172.21.15.28:0/258120665 wait complete. 2023-12-16T21:58:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 Processor -- start 2023-12-16T21:58:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- start start 2023-12-16T21:58:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c15cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f684c13f5f0 0x7f684c127730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c15adb0 0x7f684c12a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c12b190 con 0x7f684c15adb0 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c15d2c0 con 0x7f684c13f5f0 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6850cab640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c12abd0 con 0x7f684c071d90 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c15adb0 0x7f684c12a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c15adb0 0x7f684c12a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46808/0 (socket says 172.21.15.28:46808) 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 -- 172.21.15.28:0/1950863064 learned_addr learned my addr 172.21.15.28:0/1950863064 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684a59c640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c15cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f6849d9b640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f684c13f5f0 0x7f684c127730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 -- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 msgr2=0x7f684c15cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c15cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 -- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f684c13f5f0 msgr2=0x7f684c127730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f684c13f5f0 0x7f684c127730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 -- 172.21.15.28:0/1950863064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c00b6c0 con 0x7f684c15adb0 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.062+0000 7f684ad9d640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c15adb0 0x7f684c12a1f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f684400efc0 tx=0x7f684400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f683affd640 1 -- 172.21.15.28:0/1950863064 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844014480 con 0x7f684c15adb0 2023-12-16T21:58:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f683affd640 1 -- 172.21.15.28:0/1950863064 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684400f040 con 0x7f684c15adb0 2023-12-16T21:58:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684c12aeb0 con 0x7f684c15adb0 2023-12-16T21:58:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f683affd640 1 -- 172.21.15.28:0/1950863064 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844014820 con 0x7f684c15adb0 2023-12-16T21:58:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c127210 con 0x7f684c15adb0 2023-12-16T21:58:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684c126d90 con 0x7f684c15adb0 2023-12-16T21:58:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f683affd640 1 -- 172.21.15.28:0/1950863064 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6844014980 con 0x7f684c15adb0 2023-12-16T21:58:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f683affd640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f682c066470 0x7f682c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f683affd640 1 -- 172.21.15.28:0/1950863064 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f68440849c0 con 0x7f684c15adb0 2023-12-16T21:58:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f684a59c640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f682c066470 0x7f682c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.066+0000 7f684a59c640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f682c066470 0x7f682c068930 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f683c0b1000 tx=0x7f683c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.070+0000 7f683affd640 1 -- 172.21.15.28:0/1950863064 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f684c126d90 con 0x7f684c15adb0 2023-12-16T21:58:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.226+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f684c126d90 con 0x7f684c15adb0 2023-12-16T21:58:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.226+0000 7f683affd640 1 -- 172.21.15.28:0/1950863064 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f684c126d90 con 0x7f684c15adb0 2023-12-16T21:58:25.230 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f682c066470 msgr2=0x7f682c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f682c066470 0x7f682c068930 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f683c0b1000 tx=0x7f683c008040 comp rx=0 tx=0).stop 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c15adb0 msgr2=0x7f684c12a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c15adb0 0x7f684c12a1f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f684400efc0 tx=0x7f684400c490 comp rx=0 tx=0).stop 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 shutdown_connections 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f682c066470 0x7f682c068930 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f684c15adb0 0x7f684c12a1f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f684c13f5f0 0x7f684c127730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 --2- 172.21.15.28:0/1950863064 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f684c071d90 0x7f684c15cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 >> 172.21.15.28:0/1950863064 conn(0x7f684c0b0cc0 msgr2=0x7f684c1426f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 shutdown_connections 2023-12-16T21:58:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.230+0000 7f6850cab640 1 -- 172.21.15.28:0/1950863064 wait complete. 2023-12-16T21:58:25.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:25.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe637a09640 1 Processor -- start 2023-12-16T21:58:25.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe637a09640 1 -- start start 2023-12-16T21:58:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe637a09640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe6300cba10 0x7fe63015be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe637a09640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6300b42d0 0x7fe6300b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe637a09640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6300b4c60 con 0x7fe630158ca0 2023-12-16T21:58:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe637a09640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63015b670 con 0x7fe6300cba10 2023-12-16T21:58:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe637a09640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63015f410 con 0x7fe6300b42d0 2023-12-16T21:58:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe6357a5640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe630158ca0 0x7fe63015eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47526/0 (socket says 172.21.15.28:47526) 2023-12-16T21:58:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe6357a5640 1 -- 172.21.15.28:0/1904145686 learned_addr learned my addr 172.21.15.28:0/1904145686 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe634fa4640 1 --2- 172.21.15.28:0/1904145686 >> v2:172.21.15.28:3300/0 conn(0x7fe6300cba10 0x7fe63015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe635fa6640 1 --2- 172.21.15.28:0/1904145686 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6300b42d0 0x7fe6300b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe634fa4640 1 -- 172.21.15.28:0/1904145686 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6300b42d0 msgr2=0x7fe6300b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe634fa4640 1 --2- 172.21.15.28:0/1904145686 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6300b42d0 0x7fe6300b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe634fa4640 1 -- 172.21.15.28:0/1904145686 >> v1:172.21.15.28:6789/0 conn(0x7fe630158ca0 legacy=0x7fe63015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe62ffff640 1 -- 172.21.15.28:0/1904145686 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673404895 0 0) 0x7fe620001120 con 0x7fe630158ca0 2023-12-16T21:58:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.370+0000 7fe634fa4640 1 -- 172.21.15.28:0/1904145686 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63015f6f0 con 0x7fe6300cba10 2023-12-16T21:58:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe634fa4640 1 --2- 172.21.15.28:0/1904145686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300cba10 0x7fe63015be00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe6240097c0 tx=0x7fe62401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a38146566deef2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe62ffff640 1 -- 172.21.15.28:0/1904145686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6240ad640 con 0x7fe6300cba10 2023-12-16T21:58:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe62ffff640 1 -- 172.21.15.28:0/1904145686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6240ad7a0 con 0x7fe6300cba10 2023-12-16T21:58:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe62ffff640 1 -- 172.21.15.28:0/1904145686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6240adb30 con 0x7fe6300cba10 2023-12-16T21:58:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- 172.21.15.28:0/1904145686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300cba10 msgr2=0x7fe63015be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 --2- 172.21.15.28:0/1904145686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300cba10 0x7fe63015be00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe6240097c0 tx=0x7fe62401c480 comp rx=0 tx=0).stop 2023-12-16T21:58:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- 172.21.15.28:0/1904145686 shutdown_connections 2023-12-16T21:58:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 --2- 172.21.15.28:0/1904145686 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe6300b42d0 0x7fe6300b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 --2- 172.21.15.28:0/1904145686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300cba10 0x7fe63015be00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- 172.21.15.28:0/1904145686 >> 172.21.15.28:0/1904145686 conn(0x7fe6300b0cc0 msgr2=0x7fe6300b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- 172.21.15.28:0/1904145686 shutdown_connections 2023-12-16T21:58:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- 172.21.15.28:0/1904145686 wait complete. 2023-12-16T21:58:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 Processor -- start 2023-12-16T21:58:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- start start 2023-12-16T21:58:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300b42d0 0x7fe63014e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6300cba10 0x7fe63015ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe630158ca0 0x7fe630150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63014feb0 con 0x7fe6300b42d0 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63014dd60 con 0x7fe6300cba10 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe637a09640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe630152550 con 0x7fe630158ca0 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe6357a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300b42d0 0x7fe63014e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe6357a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300b42d0 0x7fe63014e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46824/0 (socket says 172.21.15.28:46824) 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe6357a5640 1 -- 172.21.15.28:0/3765490998 learned_addr learned my addr 172.21.15.28:0/3765490998 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe634fa4640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6300cba10 0x7fe63015ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe635fa6640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe630158ca0 0x7fe630150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe6357a5640 1 -- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe630158ca0 msgr2=0x7fe630150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe6357a5640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe630158ca0 0x7fe630150920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe6357a5640 1 -- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6300cba10 msgr2=0x7fe63015ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe6357a5640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6300cba10 0x7fe63015ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.374+0000 7fe6357a5640 1 -- 172.21.15.28:0/3765490998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624009420 con 0x7fe6300b42d0 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe6357a5640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300b42d0 0x7fe63014e220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c780 tx=0x7fe62000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe62dffb640 1 -- 172.21.15.28:0/3765490998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6200072d0 con 0x7fe6300b42d0 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe62dffb640 1 -- 172.21.15.28:0/3765490998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe620007450 con 0x7fe6300b42d0 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe62dffb640 1 -- 172.21.15.28:0/3765490998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6200047f0 con 0x7fe6300b42d0 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe630150120 con 0x7fe6300b42d0 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe630152830 con 0x7fe6300b42d0 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63014fd10 con 0x7fe6300b42d0 2023-12-16T21:58:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe62dffb640 1 -- 172.21.15.28:0/3765490998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe620004950 con 0x7fe6300b42d0 2023-12-16T21:58:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe62dffb640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe61c066470 0x7fe61c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe62dffb640 1 -- 172.21.15.28:0/3765490998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe6200830f0 con 0x7fe6300b42d0 2023-12-16T21:58:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe634fa4640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe61c066470 0x7fe61c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.378+0000 7fe634fa4640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe61c066470 0x7fe61c068930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe6240af000 tx=0x7fe624008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.382+0000 7fe62dffb640 1 -- 172.21.15.28:0/3765490998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe63014fd10 con 0x7fe6300b42d0 2023-12-16T21:58:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.538+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe6300b4c60 con 0x7fe6300b42d0 2023-12-16T21:58:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.538+0000 7fe62dffb640 1 -- 172.21.15.28:0/3765490998 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe6300b4c60 con 0x7fe6300b42d0 2023-12-16T21:58:25.541 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe61c066470 msgr2=0x7fe61c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe61c066470 0x7fe61c068930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe6240af000 tx=0x7fe624008040 comp rx=0 tx=0).stop 2023-12-16T21:58:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300b42d0 msgr2=0x7fe63014e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300b42d0 0x7fe63014e220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c780 tx=0x7fe62000cc40 comp rx=0 tx=0).stop 2023-12-16T21:58:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 shutdown_connections 2023-12-16T21:58:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe61c066470 0x7fe61c068930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe630158ca0 0x7fe630150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6300cba10 0x7fe63015ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 --2- 172.21.15.28:0/3765490998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6300b42d0 0x7fe63014e220 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 >> 172.21.15.28:0/3765490998 conn(0x7fe6300b0cc0 msgr2=0x7fe6300cbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 shutdown_connections 2023-12-16T21:58:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.542+0000 7fe637a09640 1 -- 172.21.15.28:0/3765490998 wait complete. 2023-12-16T21:58:25.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:58:25.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b2b0f9640 1 Processor -- start 2023-12-16T21:58:25.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b2b0f9640 1 -- start start 2023-12-16T21:58:25.682 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b2b0f9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3b24140180 0x7f3b24071f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b2b0f9640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b24071280 con 0x7f3b2406c770 2023-12-16T21:58:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b2b0f9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2406cbe0 con 0x7f3b2406c2c0 2023-12-16T21:58:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b2b0f9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b241408c0 con 0x7f3b24140180 2023-12-16T21:58:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b29696640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3b2406c770 0x7f3b24070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47540/0 (socket says 172.21.15.28:47540) 2023-12-16T21:58:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b29696640 1 -- 172.21.15.28:0/1625848393 learned_addr learned my addr 172.21.15.28:0/1625848393 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b28e95640 1 --2- 172.21.15.28:0/1625848393 >> v2:172.21.15.122:3300/0 conn(0x7f3b24140180 0x7f3b24071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200686872 0 0) 0x7f3b24071280 con 0x7f3b2406c770 2023-12-16T21:58:25.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b10003530 con 0x7f3b2406c770 2023-12-16T21:58:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2940393717 0 0) 0x7f3b10003530 con 0x7f3b2406c770 2023-12-16T21:58:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 >> v2:172.21.15.122:3300/0 conn(0x7f3b24140180 msgr2=0x7f3b24071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.678+0000 7f3b237fe640 1 --2- 172.21.15.28:0/1625848393 >> v2:172.21.15.122:3300/0 conn(0x7f3b24140180 0x7f3b24071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 >> v1:172.21.15.122:6789/0 conn(0x7f3b2406c2c0 legacy=0x7f3b24070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2415db80 con 0x7f3b2406c770 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f3b1c001680 con 0x7f3b2406c770 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 >> v1:172.21.15.28:6789/0 conn(0x7f3b2406c770 legacy=0x7f3b24070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b10003980 0x7f3b10003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b100047d0 0x7f3b10004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b1000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b241408c0 con 0x7f3b10005190 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1000c130 con 0x7f3b10003980 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1000c2f0 con 0x7f3b100047d0 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b29696640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b1000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b28e95640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b10003980 0x7f3b10003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b23fff640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b100047d0 0x7f3b10004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b29696640 1 -- 172.21.15.28:0/1625848393 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b100047d0 msgr2=0x7f3b10004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b29696640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b100047d0 0x7f3b10004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b29696640 1 -- 172.21.15.28:0/1625848393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b10003980 msgr2=0x7f3b10003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b29696640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b10003980 0x7f3b10003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b29696640 1 -- 172.21.15.28:0/1625848393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b10003530 con 0x7f3b10005190 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b29696640 1 -- 172.21.15.28:0/1625848393 reap_dead start 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b29696640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b1000b990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00c240 tx=0x7f3b1c01f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b9979b2c3d8d78 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c005ae0 con 0x7f3b10005190 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c0ab070 con 0x7f3b10005190 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b237fe640 1 -- 172.21.15.28:0/1625848393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c005ea0 con 0x7f3b10005190 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1625848393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 msgr2=0x7f3b1000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b2b0f9640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b1000b990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00c240 tx=0x7f3b1c01f3b0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1625848393 shutdown_connections 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b2b0f9640 1 --2- 172.21.15.28:0/1625848393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b1000b990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1625848393 >> 172.21.15.28:0/1625848393 conn(0x7f3b2406a530 msgr2=0x7f3b2406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1625848393 shutdown_connections 2023-12-16T21:58:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.682+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1625848393 wait complete. 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 Processor -- start 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 -- start start 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b2415c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b24158b70 0x7f3b241f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b241f23a0 0x7f3b24130950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b241318f0 con 0x7f3b10005190 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b241f2830 con 0x7f3b24158b70 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b24131610 con 0x7f3b241f23a0 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b2415c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b2415c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46838/0 (socket says 172.21.15.28:46838) 2023-12-16T21:58:25.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 -- 172.21.15.28:0/1119662462 learned_addr learned my addr 172.21.15.28:0/1119662462 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b23fff640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b24158b70 0x7f3b241f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b29696640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b241f23a0 0x7f3b24130950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 -- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b241f23a0 msgr2=0x7f3b24130950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b241f23a0 0x7f3b24130950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 -- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b24158b70 msgr2=0x7f3b241f1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b24158b70 0x7f3b241f1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 -- 172.21.15.28:0/1119662462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b24158090 con 0x7f3b10005190 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b23fff640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b24158b70 0x7f3b241f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b29696640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b241f23a0 0x7f3b24130950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b28e95640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b2415c130 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400f350 tx=0x7f3b1400f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b217fa640 1 -- 172.21.15.28:0/1119662462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14017600 con 0x7f3b10005190 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b217fa640 1 -- 172.21.15.28:0/1119662462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14011040 con 0x7f3b10005190 2023-12-16T21:58:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b217fa640 1 -- 172.21.15.28:0/1119662462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b140179c0 con 0x7f3b10005190 2023-12-16T21:58:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b24158370 con 0x7f3b10005190 2023-12-16T21:58:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2415c670 con 0x7f3b10005190 2023-12-16T21:58:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b24157ef0 con 0x7f3b10005190 2023-12-16T21:58:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.686+0000 7f3b217fa640 1 -- 172.21.15.28:0/1119662462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f3b14017b20 con 0x7f3b10005190 2023-12-16T21:58:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.690+0000 7f3b217fa640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b040664a0 0x7f3b04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.690+0000 7f3b217fa640 1 -- 172.21.15.28:0/1119662462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f3b14084670 con 0x7f3b10005190 2023-12-16T21:58:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.690+0000 7f3b23fff640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b040664a0 0x7f3b04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.690+0000 7f3b23fff640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b040664a0 0x7f3b04068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3b24158fd0 tx=0x7f3b0c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.694+0000 7f3b217fa640 1 -- 172.21.15.28:0/1119662462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3b24157ef0 con 0x7f3b10005190 2023-12-16T21:58:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3b24157ef0 con 0x7f3b10005190 2023-12-16T21:58:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b217fa640 1 -- 172.21.15.28:0/1119662462 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3b24157ef0 con 0x7f3b10005190 2023-12-16T21:58:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b040664a0 msgr2=0x7f3b04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b040664a0 0x7f3b04068960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3b24158fd0 tx=0x7f3b0c009040 comp rx=0 tx=0).stop 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 msgr2=0x7f3b2415c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b2415c130 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400f350 tx=0x7f3b1400f810 comp rx=0 tx=0).stop 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 shutdown_connections 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3b040664a0 0x7f3b04068960 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3b241f23a0 0x7f3b24130950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3b24158b70 0x7f3b241f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 --2- 172.21.15.28:0/1119662462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b10005190 0x7f3b2415c130 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 >> 172.21.15.28:0/1119662462 conn(0x7f3b2406a530 msgr2=0x7f3b24160200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:25.872 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 shutdown_connections 2023-12-16T21:58:25.872 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:25.866+0000 7f3b2b0f9640 1 -- 172.21.15.28:0/1119662462 wait complete. 2023-12-16T21:58:25.889 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-16T21:58:25.889 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:25.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.010+0000 7ffa27c26640 1 Processor -- start 2023-12-16T21:58:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.010+0000 7ffa27c26640 1 -- start start 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa27c26640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ffa20070b70 0x7ffa20165fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa27c26640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200710e0 con 0x7ffa2006c9d0 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa27c26640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200713c0 con 0x7ffa20144840 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa27c26640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2015f010 con 0x7ffa20070b70 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa251c1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffa2006c9d0 0x7ffa20070460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47552/0 (socket says 172.21.15.28:47552) 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa251c1640 1 -- 172.21.15.28:0/679964923 learned_addr learned my addr 172.21.15.28:0/679964923 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa261c3640 1 --2- 172.21.15.28:0/679964923 >> v2:172.21.15.122:3300/0 conn(0x7ffa20070b70 0x7ffa20165fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155440609 0 0) 0x7ffa200710e0 con 0x7ffa2006c9d0 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa0c003530 con 0x7ffa2006c9d0 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2809225569 0 0) 0x7ffa0c003530 con 0x7ffa2006c9d0 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 >> v2:172.21.15.122:3300/0 conn(0x7ffa20070b70 msgr2=0x7ffa20165fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 --2- 172.21.15.28:0/679964923 >> v2:172.21.15.122:3300/0 conn(0x7ffa20070b70 0x7ffa20165fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 >> v1:172.21.15.122:6789/0 conn(0x7ffa20144840 legacy=0x7ffa2006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200716a0 con 0x7ffa2006c9d0 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7ffa10001680 con 0x7ffa2006c9d0 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 >> v1:172.21.15.28:6789/0 conn(0x7ffa2006c9d0 legacy=0x7ffa20070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c003980 0x7ffa0c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa0c0047d0 0x7ffa0c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa0c005190 0x7ffa0c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c00c000 con 0x7ffa0c005190 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c00c170 con 0x7ffa0c0047d0 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0c00c330 con 0x7ffa0c003980 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa261c3640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa0c005190 0x7ffa0c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa251c1640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa0c0047d0 0x7ffa0c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa259c2640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c003980 0x7ffa0c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa261c3640 1 -- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c003980 msgr2=0x7ffa0c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa261c3640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c003980 0x7ffa0c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa261c3640 1 -- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa0c0047d0 msgr2=0x7ffa0c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa261c3640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa0c0047d0 0x7ffa0c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa261c3640 1 -- 172.21.15.28:0/679964923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c003530 con 0x7ffa0c005190 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.014+0000 7ffa251c1640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa0c0047d0 0x7ffa0c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa261c3640 1 -- 172.21.15.28:0/679964923 reap_dead start 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa261c3640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa0c005190 0x7ffa0c00b990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009fc0 tx=0x7ffa1801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f4fb05154451b04 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa180a5cb0 con 0x7ffa0c005190 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1800f040 con 0x7ffa0c005190 2023-12-16T21:58:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa249c0640 1 -- 172.21.15.28:0/679964923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18003be0 con 0x7ffa0c005190 2023-12-16T21:58:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- 172.21.15.28:0/679964923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa0c005190 msgr2=0x7ffa0c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa0c005190 0x7ffa0c00b990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009fc0 tx=0x7ffa1801cca0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- 172.21.15.28:0/679964923 shutdown_connections 2023-12-16T21:58:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 --2- 172.21.15.28:0/679964923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa0c005190 0x7ffa0c00b990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- 172.21.15.28:0/679964923 >> 172.21.15.28:0/679964923 conn(0x7ffa2006a530 msgr2=0x7ffa2006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- 172.21.15.28:0/679964923 shutdown_connections 2023-12-16T21:58:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- 172.21.15.28:0/679964923 wait complete. 2023-12-16T21:58:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 Processor -- start 2023-12-16T21:58:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- start start 2023-12-16T21:58:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c005190 0x7ffa2015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2013e5f0 0x7ffa2013ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa200c86c0 0x7ffa200c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa201f4cd0 con 0x7ffa2013e5f0 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa201f4050 con 0x7ffa200c86c0 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa27c26640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa201f4330 con 0x7ffa0c005190 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa261c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa200c86c0 0x7ffa200c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa251c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2013e5f0 0x7ffa2013ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa261c3640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa200c86c0 0x7ffa200c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:37762/0 (socket says 172.21.15.28:37762) 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa261c3640 1 -- 172.21.15.28:0/225801549 learned_addr learned my addr 172.21.15.28:0/225801549 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.018+0000 7ffa259c2640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c005190 0x7ffa2015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa251c1640 1 -- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c005190 msgr2=0x7ffa2015f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa251c1640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c005190 0x7ffa2015f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa251c1640 1 -- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa200c86c0 msgr2=0x7ffa200c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa251c1640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa200c86c0 0x7ffa200c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa251c1640 1 -- 172.21.15.28:0/225801549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa201f4610 con 0x7ffa2013e5f0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa259c2640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c005190 0x7ffa2015f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa251c1640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2013e5f0 0x7ffa2013ea70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000e550 tx=0x7ffa1000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa167fc640 1 -- 172.21.15.28:0/225801549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10012650 con 0x7ffa2013e5f0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa167fc640 1 -- 172.21.15.28:0/225801549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa100127b0 con 0x7ffa2013e5f0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa201f48f0 con 0x7ffa2013e5f0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa167fc640 1 -- 172.21.15.28:0/225801549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa10012b20 con 0x7ffa2013e5f0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa200c9080 con 0x7ffa2013e5f0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa200c9210 con 0x7ffa2013e5f0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa167fc640 1 -- 172.21.15.28:0/225801549 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7ffa10012c80 con 0x7ffa2013e5f0 2023-12-16T21:58:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa167fc640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffa04066470 0x7ffa04068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa167fc640 1 -- 172.21.15.28:0/225801549 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ffa10083bb0 con 0x7ffa2013e5f0 2023-12-16T21:58:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa259c2640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffa04066470 0x7ffa04068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.022+0000 7ffa259c2640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffa04066470 0x7ffa04068930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffa201f3f10 tx=0x7ffa1c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.026+0000 7ffa167fc640 1 -- 172.21.15.28:0/225801549 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7ffa10051030 con 0x7ffa2013e5f0 2023-12-16T21:58:26.182 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.178+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa201f4e10 con 0x7ffa2013e5f0 2023-12-16T21:58:26.184 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa167fc640 1 -- 172.21.15.28:0/225801549 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+16789 (secure 0 0 0) 0x7ffa10054650 con 0x7ffa2013e5f0 2023-12-16T21:58:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:26.184 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:24.764806+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:26.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffa04066470 msgr2=0x7ffa04068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffa04066470 0x7ffa04068930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffa201f3f10 tx=0x7ffa1c009040 comp rx=0 tx=0).stop 2023-12-16T21:58:26.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2013e5f0 msgr2=0x7ffa2013ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2013e5f0 0x7ffa2013ea70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000e550 tx=0x7ffa1000ea10 comp rx=0 tx=0).stop 2023-12-16T21:58:26.186 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 shutdown_connections 2023-12-16T21:58:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffa04066470 0x7ffa04068930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa200c86c0 0x7ffa200c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2013e5f0 0x7ffa2013ea70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 --2- 172.21.15.28:0/225801549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffa0c005190 0x7ffa2015f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 >> 172.21.15.28:0/225801549 conn(0x7ffa2006a530 msgr2=0x7ffa201422e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 shutdown_connections 2023-12-16T21:58:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.182+0000 7ffa27c26640 1 -- 172.21.15.28:0/225801549 wait complete. 2023-12-16T21:58:26.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:26.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51fb3f640 1 Processor -- start 2023-12-16T21:58:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51fb3f640 1 -- start start 2023-12-16T21:58:26.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51fb3f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd518158ca0 0x7fd51815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51fb3f640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd5180b42d0 0x7fd5180b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51fb3f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5180b4c60 con 0x7fd5180cba10 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51fb3f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd51815b670 con 0x7fd518158ca0 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51fb3f640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd51815c510 con 0x7fd5180b42d0 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51d8db640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd518158ca0 0x7fd51815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51d8db640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd518158ca0 0x7fd51815eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46872/0 (socket says 172.21.15.28:46872) 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51d8db640 1 -- 172.21.15.28:0/201730957 learned_addr learned my addr 172.21.15.28:0/201730957 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51e0dc640 1 --2- 172.21.15.28:0/201730957 >> v2:172.21.15.122:3300/0 conn(0x7fd5180b42d0 0x7fd5180b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51d8db640 1 -- 172.21.15.28:0/201730957 >> v2:172.21.15.122:3300/0 conn(0x7fd5180b42d0 msgr2=0x7fd5180b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51d8db640 1 --2- 172.21.15.28:0/201730957 >> v2:172.21.15.122:3300/0 conn(0x7fd5180b42d0 0x7fd5180b4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51d8db640 1 -- 172.21.15.28:0/201730957 >> v1:172.21.15.122:6789/0 conn(0x7fd5180cba10 legacy=0x7fd51815be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51d8db640 1 -- 172.21.15.28:0/201730957 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51815f410 con 0x7fd518158ca0 2023-12-16T21:58:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.326+0000 7fd51e0dc640 1 --2- 172.21.15.28:0/201730957 >> v2:172.21.15.122:3300/0 conn(0x7fd5180b42d0 0x7fd5180b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51d8db640 1 --2- 172.21.15.28:0/201730957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51815eb50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd5180cb970 tx=0x7fd51401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d3a5d3530c113 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51c8d9640 1 -- 172.21.15.28:0/201730957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5140a5820 con 0x7fd518158ca0 2023-12-16T21:58:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51c8d9640 1 -- 172.21.15.28:0/201730957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5140a5980 con 0x7fd518158ca0 2023-12-16T21:58:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51c8d9640 1 -- 172.21.15.28:0/201730957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5140a5d10 con 0x7fd518158ca0 2023-12-16T21:58:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- 172.21.15.28:0/201730957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 msgr2=0x7fd51815eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/201730957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51815eb50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd5180cb970 tx=0x7fd51401c960 comp rx=0 tx=0).stop 2023-12-16T21:58:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- 172.21.15.28:0/201730957 shutdown_connections 2023-12-16T21:58:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/201730957 >> v2:172.21.15.122:3300/0 conn(0x7fd5180b42d0 0x7fd5180b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/201730957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51815eb50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- 172.21.15.28:0/201730957 >> 172.21.15.28:0/201730957 conn(0x7fd5180b0cc0 msgr2=0x7fd5180b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- 172.21.15.28:0/201730957 shutdown_connections 2023-12-16T21:58:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- 172.21.15.28:0/201730957 wait complete. 2023-12-16T21:58:26.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 Processor -- start 2023-12-16T21:58:26.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- start start 2023-12-16T21:58:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5180b42d0 0x7fd518146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd5180cba10 0x7fd51815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51814d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51814e260 con 0x7fd518158ca0 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd518146f70 con 0x7fd5180b42d0 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51fb3f640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd518145f90 con 0x7fd5180cba10 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51e0dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51814d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51e0dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51814d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46880/0 (socket says 172.21.15.28:46880) 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51e0dc640 1 -- 172.21.15.28:0/2013970736 learned_addr learned my addr 172.21.15.28:0/2013970736 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51d8db640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5180b42d0 0x7fd518146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51d0da640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd5180cba10 0x7fd51815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51e0dc640 1 -- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd5180cba10 msgr2=0x7fd51815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51e0dc640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd5180cba10 0x7fd51815d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51e0dc640 1 -- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5180b42d0 msgr2=0x7fd518146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51e0dc640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5180b42d0 0x7fd518146420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.330+0000 7fd51e0dc640 1 -- 172.21.15.28:0/2013970736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd514009b30 con 0x7fd518158ca0 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd51d8db640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5180b42d0 0x7fd518146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd51e0dc640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51814d680 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd51000c820 tx=0x7fd51000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd50e7fc640 1 -- 172.21.15.28:0/2013970736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd510004240 con 0x7fd518158ca0 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd50e7fc640 1 -- 172.21.15.28:0/2013970736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5100043c0 con 0x7fd518158ca0 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd518145660 con 0x7fd518158ca0 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd50e7fc640 1 -- 172.21.15.28:0/2013970736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd510004530 con 0x7fd518158ca0 2023-12-16T21:58:26.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd518145c20 con 0x7fd518158ca0 2023-12-16T21:58:26.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd518146120 con 0x7fd518158ca0 2023-12-16T21:58:26.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd50e7fc640 1 -- 172.21.15.28:0/2013970736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fd510003860 con 0x7fd518158ca0 2023-12-16T21:58:26.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd50e7fc640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd4fc066470 0x7fd4fc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd50e7fc640 1 -- 172.21.15.28:0/2013970736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd510084180 con 0x7fd518158ca0 2023-12-16T21:58:26.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd51d8db640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd4fc066470 0x7fd4fc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.334+0000 7fd51d8db640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd4fc066470 0x7fd4fc068930 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd5140ae000 tx=0x7fd5140a7070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.338+0000 7fd50e7fc640 1 -- 172.21.15.28:0/2013970736 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fd510051570 con 0x7fd518158ca0 2023-12-16T21:58:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.494+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5180693d0 con 0x7fd518158ca0 2023-12-16T21:58:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.494+0000 7fd50e7fc640 1 -- 172.21.15.28:0/2013970736 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd510054b90 con 0x7fd518158ca0 2023-12-16T21:58:26.497 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd4fc066470 msgr2=0x7fd4fc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd4fc066470 0x7fd4fc068930 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd5140ae000 tx=0x7fd5140a7070 comp rx=0 tx=0).stop 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 msgr2=0x7fd51814d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51814d680 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd51000c820 tx=0x7fd51000cce0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 shutdown_connections 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd4fc066470 0x7fd4fc068930 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd518158ca0 0x7fd51814d680 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd5180cba10 0x7fd51815d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 --2- 172.21.15.28:0/2013970736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5180b42d0 0x7fd518146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 >> 172.21.15.28:0/2013970736 conn(0x7fd5180b0cc0 msgr2=0x7fd51815a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 shutdown_connections 2023-12-16T21:58:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.498+0000 7fd51fb3f640 1 -- 172.21.15.28:0/2013970736 wait complete. 2023-12-16T21:58:26.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:26.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb954aef640 1 Processor -- start 2023-12-16T21:58:26.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb954aef640 1 -- start start 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb954aef640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb95015d2b0 0x7fb9500d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb954aef640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb9500d02f0 0x7fb9500b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb954aef640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9500aff30 con 0x7fb9500d1cf0 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb954aef640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9500b01d0 con 0x7fb95015d2b0 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb954aef640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9500b04b0 con 0x7fb9500d02f0 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb94e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb95015d2b0 0x7fb9500d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb947fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb9500d1cf0 0x7fb9500cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47558/0 (socket says 172.21.15.28:47558) 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb947fff640 1 -- 172.21.15.28:0/1174878805 learned_addr learned my addr 172.21.15.28:0/1174878805 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb94ed9d640 1 --2- 172.21.15.28:0/1174878805 >> v2:172.21.15.122:3300/0 conn(0x7fb9500d02f0 0x7fb9500b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb94e59c640 1 -- 172.21.15.28:0/1174878805 >> v2:172.21.15.122:3300/0 conn(0x7fb9500d02f0 msgr2=0x7fb9500b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb94e59c640 1 --2- 172.21.15.28:0/1174878805 >> v2:172.21.15.122:3300/0 conn(0x7fb9500d02f0 0x7fb9500b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb94e59c640 1 -- 172.21.15.28:0/1174878805 >> v1:172.21.15.28:6789/0 conn(0x7fb9500d1cf0 legacy=0x7fb9500cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.642+0000 7fb94e59c640 1 -- 172.21.15.28:0/1174878805 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9500b0790 con 0x7fb95015d2b0 2023-12-16T21:58:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb94e59c640 1 --2- 172.21.15.28:0/1174878805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95015d2b0 0x7fb9500d17b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb9500cdb20 tx=0x7fb94001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6ed45a62b60f4e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb94dd9b640 1 -- 172.21.15.28:0/1174878805 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400ab600 con 0x7fb95015d2b0 2023-12-16T21:58:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb94dd9b640 1 -- 172.21.15.28:0/1174878805 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9400ab760 con 0x7fb95015d2b0 2023-12-16T21:58:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb94dd9b640 1 -- 172.21.15.28:0/1174878805 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400abaf0 con 0x7fb95015d2b0 2023-12-16T21:58:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 -- 172.21.15.28:0/1174878805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95015d2b0 msgr2=0x7fb9500d17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 --2- 172.21.15.28:0/1174878805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95015d2b0 0x7fb9500d17b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb9500cdb20 tx=0x7fb94001c460 comp rx=0 tx=0).stop 2023-12-16T21:58:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 -- 172.21.15.28:0/1174878805 shutdown_connections 2023-12-16T21:58:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 --2- 172.21.15.28:0/1174878805 >> v2:172.21.15.122:3300/0 conn(0x7fb9500d02f0 0x7fb9500b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 --2- 172.21.15.28:0/1174878805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb95015d2b0 0x7fb9500d17b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 -- 172.21.15.28:0/1174878805 >> 172.21.15.28:0/1174878805 conn(0x7fb95009c510 msgr2=0x7fb9500d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 -- 172.21.15.28:0/1174878805 shutdown_connections 2023-12-16T21:58:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 -- 172.21.15.28:0/1174878805 wait complete. 2023-12-16T21:58:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 Processor -- start 2023-12-16T21:58:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 -- start start 2023-12-16T21:58:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.646+0000 7fb954aef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9500d02f0 0x7fb950154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb954aef640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9500d1cf0 0x7fb95015b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb954aef640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb95015d2b0 0x7fb950155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb954aef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb950154770 con 0x7fb9500d02f0 2023-12-16T21:58:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb954aef640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015bf00 con 0x7fb9500d1cf0 2023-12-16T21:58:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb954aef640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95015a920 con 0x7fb95015d2b0 2023-12-16T21:58:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9500d02f0 0x7fb950154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9500d02f0 0x7fb950154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46902/0 (socket says 172.21.15.28:46902) 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 -- 172.21.15.28:0/3812254519 learned_addr learned my addr 172.21.15.28:0/3812254519 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb947fff640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9500d1cf0 0x7fb95015b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 -- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb95015d2b0 msgr2=0x7fb950155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb95015d2b0 0x7fb950155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 -- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9500d1cf0 msgr2=0x7fb95015b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9500d1cf0 0x7fb95015b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 -- 172.21.15.28:0/3812254519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb940009420 con 0x7fb9500d02f0 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb94e59c640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9500d02f0 0x7fb950154230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb9400ad040 tx=0x7fb94001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb946ffd640 1 -- 172.21.15.28:0/3812254519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400b5810 con 0x7fb9500d02f0 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb946ffd640 1 -- 172.21.15.28:0/3812254519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9400b5970 con 0x7fb9500d02f0 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95015ac00 con 0x7fb9500d02f0 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb946ffd640 1 -- 172.21.15.28:0/3812254519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9400b5ce0 con 0x7fb9500d02f0 2023-12-16T21:58:26.653 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95015b1c0 con 0x7fb9500d02f0 2023-12-16T21:58:26.654 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9500679f0 con 0x7fb9500d02f0 2023-12-16T21:58:26.654 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb946ffd640 1 -- 172.21.15.28:0/3812254519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fb9400b4390 con 0x7fb9500d02f0 2023-12-16T21:58:26.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb946ffd640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb930066470 0x7fb930068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb946ffd640 1 -- 172.21.15.28:0/3812254519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb94011e7d0 con 0x7fb9500d02f0 2023-12-16T21:58:26.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb947fff640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb930066470 0x7fb930068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.655 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.650+0000 7fb947fff640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb930066470 0x7fb930068930 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb95006cb50 tx=0x7fb938009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.658 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.654+0000 7fb946ffd640 1 -- 172.21.15.28:0/3812254519 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fb9400ebc50 con 0x7fb9500d02f0 2023-12-16T21:58:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb9500b1f10 con 0x7fb9500d02f0 2023-12-16T21:58:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb946ffd640 1 -- 172.21.15.28:0/3812254519 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb9400ab5c0 con 0x7fb9500d02f0 2023-12-16T21:58:26.813 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb930066470 msgr2=0x7fb930068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb930066470 0x7fb930068930 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb95006cb50 tx=0x7fb938009040 comp rx=0 tx=0).stop 2023-12-16T21:58:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9500d02f0 msgr2=0x7fb950154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9500d02f0 0x7fb950154230 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb9400ad040 tx=0x7fb94001aca0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 shutdown_connections 2023-12-16T21:58:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb930066470 0x7fb930068930 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb95015d2b0 0x7fb950155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb9500d1cf0 0x7fb95015b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 --2- 172.21.15.28:0/3812254519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9500d02f0 0x7fb950154230 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 >> 172.21.15.28:0/3812254519 conn(0x7fb95009c510 msgr2=0x7fb9500b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.810+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 shutdown_connections 2023-12-16T21:58:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.814+0000 7fb954aef640 1 -- 172.21.15.28:0/3812254519 wait complete. 2023-12-16T21:58:26.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe39d485640 1 Processor -- start 2023-12-16T21:58:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe39d485640 1 -- start start 2023-12-16T21:58:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe39d485640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe398158ca0 0x7fe39815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe39d485640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe3980cba10 0x7fe39815be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe39d485640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39815b670 con 0x7fe3980b42d0 2023-12-16T21:58:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe39d485640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39815f410 con 0x7fe398158ca0 2023-12-16T21:58:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe39d485640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39815f6f0 con 0x7fe3980cba10 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe396ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe398158ca0 0x7fe39815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe3977fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe3980b42d0 0x7fe3980703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47568/0 (socket says 172.21.15.28:47568) 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.946+0000 7fe3977fe640 1 -- 172.21.15.28:0/3750265228 learned_addr learned my addr 172.21.15.28:0/3750265228 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe3967fc640 1 --2- 172.21.15.28:0/3750265228 >> v2:172.21.15.122:3300/0 conn(0x7fe3980cba10 0x7fe39815be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe396ffd640 1 -- 172.21.15.28:0/3750265228 >> v2:172.21.15.122:3300/0 conn(0x7fe3980cba10 msgr2=0x7fe39815be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe396ffd640 1 --2- 172.21.15.28:0/3750265228 >> v2:172.21.15.122:3300/0 conn(0x7fe3980cba10 0x7fe39815be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe396ffd640 1 -- 172.21.15.28:0/3750265228 >> v1:172.21.15.28:6789/0 conn(0x7fe3980b42d0 legacy=0x7fe3980703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe396ffd640 1 -- 172.21.15.28:0/3750265228 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39815b8e0 con 0x7fe398158ca0 2023-12-16T21:58:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe395ffb640 1 -- 172.21.15.28:0/3750265228 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940939482 0 0) 0x7fe39815b670 con 0x7fe3980b42d0 2023-12-16T21:58:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe396ffd640 1 --2- 172.21.15.28:0/3750265228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe398158ca0 0x7fe39815eb50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe3980ceec0 tx=0x7fe38801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cba5ab22a47ee38 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe395ffb640 1 -- 172.21.15.28:0/3750265228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3880ab680 con 0x7fe398158ca0 2023-12-16T21:58:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe395ffb640 1 -- 172.21.15.28:0/3750265228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3880ab7e0 con 0x7fe398158ca0 2023-12-16T21:58:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe395ffb640 1 -- 172.21.15.28:0/3750265228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3880abb70 con 0x7fe398158ca0 2023-12-16T21:58:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe39d485640 1 -- 172.21.15.28:0/3750265228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe398158ca0 msgr2=0x7fe39815eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe39d485640 1 --2- 172.21.15.28:0/3750265228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe398158ca0 0x7fe39815eb50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe3980ceec0 tx=0x7fe38801c460 comp rx=0 tx=0).stop 2023-12-16T21:58:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe39d485640 1 -- 172.21.15.28:0/3750265228 shutdown_connections 2023-12-16T21:58:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe39d485640 1 --2- 172.21.15.28:0/3750265228 >> v2:172.21.15.122:3300/0 conn(0x7fe3980cba10 0x7fe39815be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe39d485640 1 --2- 172.21.15.28:0/3750265228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe398158ca0 0x7fe39815eb50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe39d485640 1 -- 172.21.15.28:0/3750265228 >> 172.21.15.28:0/3750265228 conn(0x7fe3980b0cc0 msgr2=0x7fe3980b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe39d485640 1 -- 172.21.15.28:0/3750265228 shutdown_connections 2023-12-16T21:58:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.950+0000 7fe39d485640 1 -- 172.21.15.28:0/3750265228 wait complete. 2023-12-16T21:58:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 Processor -- start 2023-12-16T21:58:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 -- start start 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3980b42d0 0x7fe39814d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3980cba10 0x7fe3980aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe398158ca0 0x7fe39813c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3980a8430 con 0x7fe3980b42d0 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39814d810 con 0x7fe3980cba10 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39814daf0 con 0x7fe398158ca0 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3980b42d0 0x7fe39814d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3980b42d0 0x7fe39814d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46928/0 (socket says 172.21.15.28:46928) 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 -- 172.21.15.28:0/4074524789 learned_addr learned my addr 172.21.15.28:0/4074524789 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe3967fc640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3980cba10 0x7fe3980aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe3977fe640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe398158ca0 0x7fe39813c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 -- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe398158ca0 msgr2=0x7fe39813c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe398158ca0 0x7fe39813c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 -- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3980cba10 msgr2=0x7fe3980aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3980cba10 0x7fe3980aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 -- 172.21.15.28:0/4074524789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388009420 con 0x7fe3980b42d0 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe396ffd640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3980b42d0 0x7fe39814d230 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe388019070 tx=0x7fe38801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe37f7fe640 1 -- 172.21.15.28:0/4074524789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3880b5810 con 0x7fe3980b42d0 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe37f7fe640 1 -- 172.21.15.28:0/4074524789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3880b5990 con 0x7fe3980b42d0 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39814ddd0 con 0x7fe3980b42d0 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe37f7fe640 1 -- 172.21.15.28:0/4074524789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3880b5d40 con 0x7fe3980b42d0 2023-12-16T21:58:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39814bba0 con 0x7fe3980b42d0 2023-12-16T21:58:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39814dc30 con 0x7fe3980b42d0 2023-12-16T21:58:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.954+0000 7fe37f7fe640 1 -- 172.21.15.28:0/4074524789 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe3880bf410 con 0x7fe3980b42d0 2023-12-16T21:58:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.958+0000 7fe37f7fe640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe3840664a0 0x7fe384068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.958+0000 7fe37f7fe640 1 -- 172.21.15.28:0/4074524789 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe38811eb40 con 0x7fe3980b42d0 2023-12-16T21:58:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.958+0000 7fe3967fc640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe3840664a0 0x7fe384068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.958+0000 7fe3967fc640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe3840664a0 0x7fe384068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe3980b4e90 tx=0x7fe38c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:26.962+0000 7fe37f7fe640 1 -- 172.21.15.28:0/4074524789 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe39814dc30 con 0x7fe3980b42d0 2023-12-16T21:58:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.110+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe3980a88c0 con 0x7fe3980b42d0 2023-12-16T21:58:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.110+0000 7fe37f7fe640 1 -- 172.21.15.28:0/4074524789 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe3980a88c0 con 0x7fe3980b42d0 2023-12-16T21:58:27.113 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe3840664a0 msgr2=0x7fe384068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe3840664a0 0x7fe384068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe3980b4e90 tx=0x7fe38c008040 comp rx=0 tx=0).stop 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3980b42d0 msgr2=0x7fe39814d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3980b42d0 0x7fe39814d230 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe388019070 tx=0x7fe38801aca0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 shutdown_connections 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe3840664a0 0x7fe384068960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe398158ca0 0x7fe39813c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3980cba10 0x7fe3980aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 --2- 172.21.15.28:0/4074524789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3980b42d0 0x7fe39814d230 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 >> 172.21.15.28:0/4074524789 conn(0x7fe3980b0cc0 msgr2=0x7fe3980b10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 shutdown_connections 2023-12-16T21:58:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.114+0000 7fe39d485640 1 -- 172.21.15.28:0/4074524789 wait complete. 2023-12-16T21:58:27.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:58:27.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb085ac640 1 Processor -- start 2023-12-16T21:58:27.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb085ac640 1 -- start start 2023-12-16T21:58:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb085ac640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffb0015d160 0x7ffb000aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb085ac640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 0x7ffb000ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb085ac640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb000addf0 con 0x7ffb0006b540 2023-12-16T21:58:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb085ac640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0006e010 con 0x7ffb0015d160 2023-12-16T21:58:27.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb085ac640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb000aea70 con 0x7ffb0006cd80 2023-12-16T21:58:27.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb06b49640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffb0006b540 0x7ffb000b27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47582/0 (socket says 172.21.15.28:47582) 2023-12-16T21:58:27.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb06348640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffb0015d160 0x7ffb000aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb06b49640 1 -- 172.21.15.28:0/882846396 learned_addr learned my addr 172.21.15.28:0/882846396 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:27.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb05b47640 1 --2- 172.21.15.28:0/882846396 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 0x7ffb000ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb06348640 1 -- 172.21.15.28:0/882846396 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 msgr2=0x7ffb000ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:27.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb06348640 1 --2- 172.21.15.28:0/882846396 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 0x7ffb000ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb06348640 1 -- 172.21.15.28:0/882846396 >> v1:172.21.15.28:6789/0 conn(0x7ffb0006b540 legacy=0x7ffb000b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb06348640 1 -- 172.21.15.28:0/882846396 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb000ae0d0 con 0x7ffb0015d160 2023-12-16T21:58:27.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.238+0000 7ffb05b47640 1 --2- 172.21.15.28:0/882846396 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 0x7ffb000ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:27.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb06348640 1 --2- 172.21.15.28:0/882846396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0015d160 0x7ffb000aee90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009d90 tx=0x7ffafc01c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76da8c61b1047c7d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb05346640 1 -- 172.21.15.28:0/882846396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0a5770 con 0x7ffb0015d160 2023-12-16T21:58:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb05346640 1 -- 172.21.15.28:0/882846396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc0a58d0 con 0x7ffb0015d160 2023-12-16T21:58:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb05346640 1 -- 172.21.15.28:0/882846396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0a5c60 con 0x7ffb0015d160 2023-12-16T21:58:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb085ac640 1 -- 172.21.15.28:0/882846396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0015d160 msgr2=0x7ffb000aee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb085ac640 1 --2- 172.21.15.28:0/882846396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0015d160 0x7ffb000aee90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009d90 tx=0x7ffafc01c750 comp rx=0 tx=0).stop 2023-12-16T21:58:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb085ac640 1 -- 172.21.15.28:0/882846396 shutdown_connections 2023-12-16T21:58:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb085ac640 1 --2- 172.21.15.28:0/882846396 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 0x7ffb000ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb085ac640 1 --2- 172.21.15.28:0/882846396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0015d160 0x7ffb000aee90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb085ac640 1 -- 172.21.15.28:0/882846396 >> 172.21.15.28:0/882846396 conn(0x7ffb000b1af0 msgr2=0x7ffb00070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb085ac640 1 -- 172.21.15.28:0/882846396 shutdown_connections 2023-12-16T21:58:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.242+0000 7ffb085ac640 1 -- 172.21.15.28:0/882846396 wait complete. 2023-12-16T21:58:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 Processor -- start 2023-12-16T21:58:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 -- start start 2023-12-16T21:58:27.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006b540 0x7ffb0015f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 0x7ffb001520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb0015d160 0x7ffb001525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb001539b0 con 0x7ffb0006b540 2023-12-16T21:58:27.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb00152e30 con 0x7ffb0015d160 2023-12-16T21:58:27.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb00153110 con 0x7ffb0006cd80 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006b540 0x7ffb0015f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006b540 0x7ffb0015f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46954/0 (socket says 172.21.15.28:46954) 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 -- 172.21.15.28:0/4122802216 learned_addr learned my addr 172.21.15.28:0/4122802216 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 -- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 msgr2=0x7ffb001520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 0x7ffb001520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 -- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb0015d160 msgr2=0x7ffb001525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb0015d160 0x7ffb001525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 -- 172.21.15.28:0/4122802216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc009a70 con 0x7ffb0006b540 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb06348640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006b540 0x7ffb0015f9c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffafc01cff0 tx=0x7ffafc0194f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffaf6ffd640 1 -- 172.21.15.28:0/4122802216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0aec90 con 0x7ffb0006b540 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffaf6ffd640 1 -- 172.21.15.28:0/4122802216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc0aedf0 con 0x7ffb0006b540 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb001533f0 con 0x7ffb0006b540 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffaf6ffd640 1 -- 172.21.15.28:0/4122802216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0b7650 con 0x7ffb0006b540 2023-12-16T21:58:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb00152b20 con 0x7ffb0006b540 2023-12-16T21:58:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0015d870 con 0x7ffb0006b540 2023-12-16T21:58:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffaf6ffd640 1 -- 172.21.15.28:0/4122802216 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7ffafc0b4050 con 0x7ffb0006b540 2023-12-16T21:58:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.246+0000 7ffaf6ffd640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffae4066470 0x7ffae4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.250+0000 7ffaf6ffd640 1 -- 172.21.15.28:0/4122802216 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ffafc11f650 con 0x7ffb0006b540 2023-12-16T21:58:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.250+0000 7ffb05b47640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffae4066470 0x7ffae4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.252 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.250+0000 7ffb05b47640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffae4066470 0x7ffae4068930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffb00151980 tx=0x7ffaf0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.250+0000 7ffaf6ffd640 1 -- 172.21.15.28:0/4122802216 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7ffafc0eccd0 con 0x7ffb0006b540 2023-12-16T21:58:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffb00153250 con 0x7ffb0006b540 2023-12-16T21:58:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffaf6ffd640 1 -- 172.21.15.28:0/4122802216 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffb00153250 con 0x7ffb0006b540 2023-12-16T21:58:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2023-12-16T21:58:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffae4066470 msgr2=0x7ffae4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffae4066470 0x7ffae4068930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffb00151980 tx=0x7ffaf0008040 comp rx=0 tx=0).stop 2023-12-16T21:58:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006b540 msgr2=0x7ffb0015f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006b540 0x7ffb0015f9c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffafc01cff0 tx=0x7ffafc0194f0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 shutdown_connections 2023-12-16T21:58:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ffae4066470 0x7ffae4068930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb0015d160 0x7ffb001525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ffb0006cd80 0x7ffb001520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 --2- 172.21.15.28:0/4122802216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006b540 0x7ffb0015f9c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 >> 172.21.15.28:0/4122802216 conn(0x7ffb000b1af0 msgr2=0x7ffb000b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:27.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 shutdown_connections 2023-12-16T21:58:27.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.418+0000 7ffb085ac640 1 -- 172.21.15.28:0/4122802216 wait complete. 2023-12-16T21:58:27.438 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi122.front.sepia.ceph.com... 2023-12-16T21:58:27.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:27.438 DEBUG:teuthology.orchestra.run.smithi122:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-12-16T21:58:27.453 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:58:27.453 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2023-12-16T21:58:27.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:27.453 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-16T21:58:27.502 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-16T21:58:27.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:58:27.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.622+0000 7f76a5b36640 1 Processor -- start 2023-12-16T21:58:27.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.622+0000 7f76a5b36640 1 -- start start 2023-12-16T21:58:27.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f76a5b36640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a0070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f76a5b36640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76a0072180 0x7f76a00709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f76a5b36640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a00710e0 con 0x7f76a0144840 2023-12-16T21:58:27.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f76a5b36640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a015f010 con 0x7f76a0072180 2023-12-16T21:58:27.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f76a5b36640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a0070ee0 con 0x7f76a006c9d0 2023-12-16T21:58:27.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769ffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76a0072180 0x7f76a00709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769ffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76a0072180 0x7f76a00709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48588/0 (socket says 172.21.15.28:48588) 2023-12-16T21:58:27.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769f7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f76a0144840 0x7f76a006c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51984/0 (socket says 172.21.15.28:51984) 2023-12-16T21:58:27.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769ffff640 1 -- 172.21.15.28:0/1539275924 learned_addr learned my addr 172.21.15.28:0/1539275924 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:27.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f7696ffd640 1 --2- 172.21.15.28:0/1539275924 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a0070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769ffff640 1 -- 172.21.15.28:0/1539275924 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 msgr2=0x7f76a0070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:27.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769ffff640 1 --2- 172.21.15.28:0/1539275924 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a0070460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769ffff640 1 -- 172.21.15.28:0/1539275924 >> v1:172.21.15.28:6789/0 conn(0x7f76a0144840 legacy=0x7f76a006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769ffff640 1 -- 172.21.15.28:0/1539275924 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a00713c0 con 0x7f76a0072180 2023-12-16T21:58:27.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769effd640 1 -- 172.21.15.28:0/1539275924 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563548945 0 0) 0x7f76a00710e0 con 0x7f76a0144840 2023-12-16T21:58:27.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f7696ffd640 1 --2- 172.21.15.28:0/1539275924 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a0070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769ffff640 1 --2- 172.21.15.28:0/1539275924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00709a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7698009e50 tx=0x7f769801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566c53826a0d4a78 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769effd640 1 -- 172.21.15.28:0/1539275924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76980a58a0 con 0x7f76a0072180 2023-12-16T21:58:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769effd640 1 -- 172.21.15.28:0/1539275924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76980a5a00 con 0x7f76a0072180 2023-12-16T21:58:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.626+0000 7f769effd640 1 -- 172.21.15.28:0/1539275924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76980a5d90 con 0x7f76a0072180 2023-12-16T21:58:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- 172.21.15.28:0/1539275924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 msgr2=0x7f76a00709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1539275924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00709a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7698009e50 tx=0x7f769801c9e0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- 172.21.15.28:0/1539275924 shutdown_connections 2023-12-16T21:58:27.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1539275924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00709a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1539275924 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a0070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- 172.21.15.28:0/1539275924 >> 172.21.15.28:0/1539275924 conn(0x7f76a006a530 msgr2=0x7f76a006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- 172.21.15.28:0/1539275924 shutdown_connections 2023-12-16T21:58:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- 172.21.15.28:0/1539275924 wait complete. 2023-12-16T21:58:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 Processor -- start 2023-12-16T21:58:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- start start 2023-12-16T21:58:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a015ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a0144840 0x7f76a00b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a00b3890 con 0x7f76a0072180 2023-12-16T21:58:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a0143d60 con 0x7f76a0144840 2023-12-16T21:58:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a00b2c80 con 0x7f76a006c9d0 2023-12-16T21:58:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00b20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48594/0 (socket says 172.21.15.28:48594) 2023-12-16T21:58:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 -- 172.21.15.28:0/1600889276 learned_addr learned my addr 172.21.15.28:0/1600889276 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f769f7fe640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a015ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 -- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 msgr2=0x7f76a015ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a015ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 -- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a0144840 msgr2=0x7f76a00b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f769ffff640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a0144840 0x7f76a00b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a0144840 0x7f76a00b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 -- 172.21.15.28:0/1600889276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698009b30 con 0x7f76a0072180 2023-12-16T21:58:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f7696ffd640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00b20c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f768800c7f0 tx=0x7f768800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f769cff9640 1 -- 172.21.15.28:0/1600889276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768800cea0 con 0x7f76a0072180 2023-12-16T21:58:27.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f769cff9640 1 -- 172.21.15.28:0/1600889276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7688007a30 con 0x7f76a0072180 2023-12-16T21:58:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a00b2f60 con 0x7f76a0072180 2023-12-16T21:58:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f769cff9640 1 -- 172.21.15.28:0/1600889276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688007dc0 con 0x7f76a0072180 2023-12-16T21:58:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.630+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a00b3520 con 0x7f76a0072180 2023-12-16T21:58:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.634+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a00cd460 con 0x7f76a0072180 2023-12-16T21:58:27.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.634+0000 7f769cff9640 1 -- 172.21.15.28:0/1600889276 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7688024480 con 0x7f76a0072180 2023-12-16T21:58:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.634+0000 7f769cff9640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7680066470 0x7f7680068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.634+0000 7f769cff9640 1 -- 172.21.15.28:0/1600889276 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7688084430 con 0x7f76a0072180 2023-12-16T21:58:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.634+0000 7f769f7fe640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7680066470 0x7f7680068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.634+0000 7f769f7fe640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7680066470 0x7f7680068930 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7690000f70 tx=0x7f7690009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.638+0000 7f769cff9640 1 -- 172.21.15.28:0/1600889276 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7688051900 con 0x7f76a0072180 2023-12-16T21:58:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.826+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76a0069440 con 0x7f76a0072180 2023-12-16T21:58:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.826+0000 7f769cff9640 1 -- 172.21.15.28:0/1600889276 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4932 (secure 0 0 0) 0x7f7688054f20 con 0x7f76a0072180 2023-12-16T21:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:27.831 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:57:42.850798+0000","modified":"2023-12-16T21:57:45.892479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:58:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2023-12-16T21:58:27.833 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7680066470 msgr2=0x7f7680068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.833 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7680066470 0x7f7680068930 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7690000f70 tx=0x7f7690009040 comp rx=0 tx=0).stop 2023-12-16T21:58:27.833 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 msgr2=0x7f76a00b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.833 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00b20c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f768800c7f0 tx=0x7f768800ccb0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.833 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 shutdown_connections 2023-12-16T21:58:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7680066470 0x7f7680068930 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76a0144840 0x7f76a00b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a0072180 0x7f76a00b20c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 --2- 172.21.15.28:0/1600889276 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f76a006c9d0 0x7f76a015ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 >> 172.21.15.28:0/1600889276 conn(0x7f76a006a530 msgr2=0x7f76a015dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 shutdown_connections 2023-12-16T21:58:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.830+0000 7f76a5b36640 1 -- 172.21.15.28:0/1600889276 wait complete. 2023-12-16T21:58:27.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.970+0000 7f0419b2a640 1 Processor -- start 2023-12-16T21:58:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.970+0000 7f0419b2a640 1 -- start start 2023-12-16T21:58:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.970+0000 7f0419b2a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f041413cf30 0x7f041406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.970+0000 7f0419b2a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f041406c800 0x7f041415a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.970+0000 7f0419b2a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 0x7f0414071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.970+0000 7f0419b2a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041413d310 con 0x7f041413cf30 2023-12-16T21:58:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.970+0000 7f0419b2a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04141404e0 con 0x7f041406c800 2023-12-16T21:58:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.970+0000 7f0419b2a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041406cc70 con 0x7f041415ab70 2023-12-16T21:58:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f041413cf30 0x7f041406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f041413cf30 0x7f041406c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48608/0 (socket says 172.21.15.28:48608) 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 -- 172.21.15.28:0/3162143235 learned_addr learned my addr 172.21.15.28:0/3162143235 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0412ffd640 1 --2- 172.21.15.28:0/3162143235 >> v2:172.21.15.122:3300/0 conn(0x7f041406c800 0x7f041415a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0413fff640 1 --2- 172.21.15.28:0/3162143235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 0x7f0414071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 -- 172.21.15.28:0/3162143235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 msgr2=0x7f0414071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 --2- 172.21.15.28:0/3162143235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 0x7f0414071570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 -- 172.21.15.28:0/3162143235 >> v2:172.21.15.122:3300/0 conn(0x7f041406c800 msgr2=0x7f041415a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 --2- 172.21.15.28:0/3162143235 >> v2:172.21.15.122:3300/0 conn(0x7f041406c800 0x7f041415a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 -- 172.21.15.28:0/3162143235 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041415b3a0 con 0x7f041413cf30 2023-12-16T21:58:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0412ffd640 1 --2- 172.21.15.28:0/3162143235 >> v2:172.21.15.122:3300/0 conn(0x7f041406c800 0x7f041415a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04137fe640 1 --2- 172.21.15.28:0/3162143235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041413cf30 0x7f041406c2c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f04140cc790 tx=0x7f040801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47c36ff9c1060dc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04127fc640 1 -- 172.21.15.28:0/3162143235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04080ac640 con 0x7f041413cf30 2023-12-16T21:58:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04127fc640 1 -- 172.21.15.28:0/3162143235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04080ac7a0 con 0x7f041413cf30 2023-12-16T21:58:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f04127fc640 1 -- 172.21.15.28:0/3162143235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04080acb30 con 0x7f041413cf30 2023-12-16T21:58:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 -- 172.21.15.28:0/3162143235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041413cf30 msgr2=0x7f041406c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 --2- 172.21.15.28:0/3162143235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041413cf30 0x7f041406c2c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f04140cc790 tx=0x7f040801c420 comp rx=0 tx=0).stop 2023-12-16T21:58:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 -- 172.21.15.28:0/3162143235 shutdown_connections 2023-12-16T21:58:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 --2- 172.21.15.28:0/3162143235 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 0x7f0414071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 --2- 172.21.15.28:0/3162143235 >> v2:172.21.15.122:3300/0 conn(0x7f041406c800 0x7f041415a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 --2- 172.21.15.28:0/3162143235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041413cf30 0x7f041406c2c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 -- 172.21.15.28:0/3162143235 >> 172.21.15.28:0/3162143235 conn(0x7f041406a530 msgr2=0x7f041406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 -- 172.21.15.28:0/3162143235 shutdown_connections 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.974+0000 7f0419b2a640 1 -- 172.21.15.28:0/3162143235 wait complete. 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 Processor -- start 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 -- start start 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041406c800 0x7f041415cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f041413cf30 0x7f04140afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 0x7f04140aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04141566f0 con 0x7f041406c800 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04141569d0 con 0x7f041413cf30 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0414156cb0 con 0x7f041415ab70 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041406c800 0x7f041415cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041406c800 0x7f041415cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48622/0 (socket says 172.21.15.28:48622) 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 -- 172.21.15.28:0/2337007703 learned_addr learned my addr 172.21.15.28:0/2337007703 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0413fff640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 0x7f04140aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0412ffd640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f041413cf30 0x7f04140afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 -- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 msgr2=0x7f04140aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 0x7f04140aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 -- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f041413cf30 msgr2=0x7f04140afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f041413cf30 0x7f04140afa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 -- 172.21.15.28:0/2337007703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04080094d0 con 0x7f041406c800 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f04137fe640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041406c800 0x7f041415cca0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f040801aad0 tx=0x7f040801ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f03fbfff640 1 -- 172.21.15.28:0/2337007703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04080b7860 con 0x7f041406c800 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f03fbfff640 1 -- 172.21.15.28:0/2337007703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04080b79e0 con 0x7f041406c800 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0414156f90 con 0x7f041406c800 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f03fbfff640 1 -- 172.21.15.28:0/2337007703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04080b7d90 con 0x7f041406c800 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04140aedc0 con 0x7f041406c800 2023-12-16T21:58:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04140af070 con 0x7f041406c800 2023-12-16T21:58:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f03fbfff640 1 -- 172.21.15.28:0/2337007703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f04080b3070 con 0x7f041406c800 2023-12-16T21:58:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.978+0000 7f03fbfff640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0400066470 0x7f0400068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.982+0000 7f03fbfff640 1 -- 172.21.15.28:0/2337007703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f040811e9b0 con 0x7f041406c800 2023-12-16T21:58:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.982+0000 7f0412ffd640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0400066470 0x7f0400068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.982+0000 7f0412ffd640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0400066470 0x7f0400068930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0404006fd0 tx=0x7f0404008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:27.986+0000 7f03fbfff640 1 -- 172.21.15.28:0/2337007703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f04140af070 con 0x7f041406c800 2023-12-16T21:58:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.142+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04140af070 con 0x7f041406c800 2023-12-16T21:58:28.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f03fbfff640 1 -- 172.21.15.28:0/2337007703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f04140af070 con 0x7f041406c800 2023-12-16T21:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:28.149 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:24.764806+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0400066470 msgr2=0x7f0400068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0400066470 0x7f0400068930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0404006fd0 tx=0x7f0404008040 comp rx=0 tx=0).stop 2023-12-16T21:58:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041406c800 msgr2=0x7f041415cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041406c800 0x7f041415cca0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f040801aad0 tx=0x7f040801ac60 comp rx=0 tx=0).stop 2023-12-16T21:58:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 shutdown_connections 2023-12-16T21:58:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0400066470 0x7f0400068930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f041415ab70 0x7f04140aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f041413cf30 0x7f04140afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 --2- 172.21.15.28:0/2337007703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f041406c800 0x7f041415cca0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 >> 172.21.15.28:0/2337007703 conn(0x7f041406a530 msgr2=0x7f04140cd910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 shutdown_connections 2023-12-16T21:58:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.146+0000 7f0419b2a640 1 -- 172.21.15.28:0/2337007703 wait complete. 2023-12-16T21:58:28.166 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:57:41.198086+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:57:41.842692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:28.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ef092640 1 Processor -- start 2023-12-16T21:58:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ef092640 1 -- start start 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ef092640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f61e80cba10 0x7f61e815be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ef092640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80b42d0 0x7f61e80b4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ef092640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e80b4c60 con 0x7f61e8158ca0 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ef092640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e815b670 con 0x7f61e80cba10 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ef092640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e815f410 con 0x7f61e80b42d0 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ece2e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f61e8158ca0 0x7f61e815eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51994/0 (socket says 172.21.15.28:51994) 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61e7fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f61e80cba10 0x7f61e815be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ece2e640 1 -- 172.21.15.28:0/2409460358 learned_addr learned my addr 172.21.15.28:0/2409460358 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61ed62f640 1 --2- 172.21.15.28:0/2409460358 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80b42d0 0x7f61e80b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61e7fff640 1 -- 172.21.15.28:0/2409460358 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80b42d0 msgr2=0x7f61e80b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:28.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61e7fff640 1 --2- 172.21.15.28:0/2409460358 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80b42d0 0x7f61e80b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61e7fff640 1 -- 172.21.15.28:0/2409460358 >> v1:172.21.15.28:6789/0 conn(0x7f61e8158ca0 legacy=0x7f61e815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61e7fff640 1 -- 172.21.15.28:0/2409460358 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e815f6f0 con 0x7f61e80cba10 2023-12-16T21:58:28.295 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.290+0000 7f61e77fe640 1 -- 172.21.15.28:0/2409460358 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418412494 0 0) 0x7f61dc001120 con 0x7f61e8158ca0 2023-12-16T21:58:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61e7fff640 1 --2- 172.21.15.28:0/2409460358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80cba10 0x7f61e815be00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f61d40097c0 tx=0x7f61d401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631ecc89e9932a29 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61e77fe640 1 -- 172.21.15.28:0/2409460358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40ad640 con 0x7f61e80cba10 2023-12-16T21:58:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61e77fe640 1 -- 172.21.15.28:0/2409460358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d40ad7a0 con 0x7f61e80cba10 2023-12-16T21:58:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61e77fe640 1 -- 172.21.15.28:0/2409460358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40adb30 con 0x7f61e80cba10 2023-12-16T21:58:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- 172.21.15.28:0/2409460358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80cba10 msgr2=0x7f61e815be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 --2- 172.21.15.28:0/2409460358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80cba10 0x7f61e815be00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f61d40097c0 tx=0x7f61d401c480 comp rx=0 tx=0).stop 2023-12-16T21:58:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- 172.21.15.28:0/2409460358 shutdown_connections 2023-12-16T21:58:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 --2- 172.21.15.28:0/2409460358 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80b42d0 0x7f61e80b4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 --2- 172.21.15.28:0/2409460358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80cba10 0x7f61e815be00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- 172.21.15.28:0/2409460358 >> 172.21.15.28:0/2409460358 conn(0x7f61e80b0cc0 msgr2=0x7f61e80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- 172.21.15.28:0/2409460358 shutdown_connections 2023-12-16T21:58:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- 172.21.15.28:0/2409460358 wait complete. 2023-12-16T21:58:28.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 Processor -- start 2023-12-16T21:58:28.298 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- start start 2023-12-16T21:58:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80b42d0 0x7f61e814f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80cba10 0x7f61e8154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61e8158ca0 0x7f61e81552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e814edc0 con 0x7f61e80b42d0 2023-12-16T21:58:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e8153a70 con 0x7f61e8158ca0 2023-12-16T21:58:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ef092640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e81534c0 con 0x7f61e80cba10 2023-12-16T21:58:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ece2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80b42d0 0x7f61e814f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ece2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80b42d0 0x7f61e814f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48640/0 (socket says 172.21.15.28:48640) 2023-12-16T21:58:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ece2e640 1 -- 172.21.15.28:0/1838104577 learned_addr learned my addr 172.21.15.28:0/1838104577 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61e7fff640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80cba10 0x7f61e8154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ed62f640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61e8158ca0 0x7f61e81552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ece2e640 1 -- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80cba10 msgr2=0x7f61e8154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.294+0000 7f61ece2e640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80cba10 0x7f61e8154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61ece2e640 1 -- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61e8158ca0 msgr2=0x7f61e81552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61ece2e640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61e8158ca0 0x7f61e81552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61ece2e640 1 -- 172.21.15.28:0/1838104577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d4009420 con 0x7f61e80b42d0 2023-12-16T21:58:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61ece2e640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80b42d0 0x7f61e814f250 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00c780 tx=0x7f61dc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61e57fa640 1 -- 172.21.15.28:0/1838104577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc0072d0 con 0x7f61e80b42d0 2023-12-16T21:58:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61e57fa640 1 -- 172.21.15.28:0/1838104577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc007450 con 0x7f61e80b42d0 2023-12-16T21:58:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e8153790 con 0x7f61e80b42d0 2023-12-16T21:58:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61e57fa640 1 -- 172.21.15.28:0/1838104577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc0047f0 con 0x7f61e80b42d0 2023-12-16T21:58:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e814ea50 con 0x7f61e80b42d0 2023-12-16T21:58:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e80693d0 con 0x7f61e80b42d0 2023-12-16T21:58:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61e57fa640 1 -- 172.21.15.28:0/1838104577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f61dc004950 con 0x7f61e80b42d0 2023-12-16T21:58:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61e57fa640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f61cc066470 0x7f61cc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61e57fa640 1 -- 172.21.15.28:0/1838104577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f61dc083230 con 0x7f61e80b42d0 2023-12-16T21:58:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61e7fff640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f61cc066470 0x7f61cc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.298+0000 7f61e7fff640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f61cc066470 0x7f61cc068930 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f61d40af000 tx=0x7f61d40a7070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.302+0000 7f61e57fa640 1 -- 172.21.15.28:0/1838104577 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f61dc0506b0 con 0x7f61e80b42d0 2023-12-16T21:58:28.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.450+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61e80b4c60 con 0x7f61e80b42d0 2023-12-16T21:58:28.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.450+0000 7f61e57fa640 1 -- 172.21.15.28:0/1838104577 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61e80b4c60 con 0x7f61e80b42d0 2023-12-16T21:58:28.455 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f61cc066470 msgr2=0x7f61cc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f61cc066470 0x7f61cc068930 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f61d40af000 tx=0x7f61d40a7070 comp rx=0 tx=0).stop 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80b42d0 msgr2=0x7f61e814f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80b42d0 0x7f61e814f250 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00c780 tx=0x7f61dc00cc40 comp rx=0 tx=0).stop 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 shutdown_connections 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f61cc066470 0x7f61cc068930 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f61e8158ca0 0x7f61e81552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f61e80cba10 0x7f61e8154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 --2- 172.21.15.28:0/1838104577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e80b42d0 0x7f61e814f250 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 >> 172.21.15.28:0/1838104577 conn(0x7f61e80b0cc0 msgr2=0x7f61e80b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 shutdown_connections 2023-12-16T21:58:28.458 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.454+0000 7f61ef092640 1 -- 172.21.15.28:0/1838104577 wait complete. 2023-12-16T21:58:28.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:58:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244e6b8640 1 Processor -- start 2023-12-16T21:58:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244e6b8640 1 -- start start 2023-12-16T21:58:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244e6b8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f244815d2b0 0x7f24480d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244e6b8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24480d1cf0 0x7f24480cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244e6b8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24480aff30 con 0x7f244815d690 2023-12-16T21:58:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244e6b8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24480b01d0 con 0x7f24480d1cf0 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244e6b8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24480b04b0 con 0x7f244815d2b0 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244cc55640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f244815d690 0x7f24480cdd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52002/0 (socket says 172.21.15.28:52002) 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f24477fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24480d1cf0 0x7f24480cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f244cc55640 1 -- 172.21.15.28:0/524147145 learned_addr learned my addr 172.21.15.28:0/524147145 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f2447fff640 1 --2- 172.21.15.28:0/524147145 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f244815d2b0 0x7f24480d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f24477fe640 1 -- 172.21.15.28:0/524147145 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f244815d2b0 msgr2=0x7f24480d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f24477fe640 1 --2- 172.21.15.28:0/524147145 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f244815d2b0 0x7f24480d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f24477fe640 1 -- 172.21.15.28:0/524147145 >> v1:172.21.15.28:6789/0 conn(0x7f244815d690 legacy=0x7f24480cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f2446ffd640 1 -- 172.21.15.28:0/524147145 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817464388 0 0) 0x7f24480aff30 con 0x7f244815d690 2023-12-16T21:58:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.590+0000 7f24477fe640 1 -- 172.21.15.28:0/524147145 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24480b0790 con 0x7f24480d1cf0 2023-12-16T21:58:28.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f24477fe640 1 --2- 172.21.15.28:0/524147145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24480d1cf0 0x7f24480cfbe0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f243c0097c0 tx=0x7f243c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e068d23881e36628 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f2446ffd640 1 -- 172.21.15.28:0/524147145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c0ab620 con 0x7f24480d1cf0 2023-12-16T21:58:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f2446ffd640 1 -- 172.21.15.28:0/524147145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243c0ab780 con 0x7f24480d1cf0 2023-12-16T21:58:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f2446ffd640 1 -- 172.21.15.28:0/524147145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c0abb10 con 0x7f24480d1cf0 2023-12-16T21:58:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- 172.21.15.28:0/524147145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24480d1cf0 msgr2=0x7f24480cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 --2- 172.21.15.28:0/524147145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24480d1cf0 0x7f24480cfbe0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f243c0097c0 tx=0x7f243c01c480 comp rx=0 tx=0).stop 2023-12-16T21:58:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- 172.21.15.28:0/524147145 shutdown_connections 2023-12-16T21:58:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 --2- 172.21.15.28:0/524147145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24480d1cf0 0x7f24480cfbe0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 --2- 172.21.15.28:0/524147145 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f244815d2b0 0x7f24480d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- 172.21.15.28:0/524147145 >> 172.21.15.28:0/524147145 conn(0x7f244809c510 msgr2=0x7f24480d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- 172.21.15.28:0/524147145 shutdown_connections 2023-12-16T21:58:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- 172.21.15.28:0/524147145 wait complete. 2023-12-16T21:58:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 Processor -- start 2023-12-16T21:58:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- start start 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f24480d1cf0 0x7f2448152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f244815d2b0 0x7f24481534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244815d690 0x7f244815a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2448154d30 con 0x7f244815d690 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2448155010 con 0x7f244815d2b0 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244e6b8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24481552f0 con 0x7f24480d1cf0 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244cc55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244815d690 0x7f244815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244cc55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244815d690 0x7f244815a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48658/0 (socket says 172.21.15.28:48658) 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244cc55640 1 -- 172.21.15.28:0/3333888073 learned_addr learned my addr 172.21.15.28:0/3333888073 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f2447fff640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f24480d1cf0 0x7f2448152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f24477fe640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f244815d2b0 0x7f24481534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244cc55640 1 -- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f24480d1cf0 msgr2=0x7f2448152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244cc55640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f24480d1cf0 0x7f2448152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244cc55640 1 -- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f244815d2b0 msgr2=0x7f24481534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244cc55640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f244815d2b0 0x7f24481534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f244cc55640 1 -- 172.21.15.28:0/3333888073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c009420 con 0x7f244815d690 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f2447fff640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f24480d1cf0 0x7f2448152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.594+0000 7f24477fe640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f244815d2b0 0x7f24481534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f244cc55640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244815d690 0x7f244815a920 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f244000c780 tx=0x7f244000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f2444ff9640 1 -- 172.21.15.28:0/3333888073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244000ce90 con 0x7f244815d690 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f2444ff9640 1 -- 172.21.15.28:0/3333888073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244000a640 con 0x7f244815d690 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24481555d0 con 0x7f244815d690 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f2444ff9640 1 -- 172.21.15.28:0/3333888073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244000a9d0 con 0x7f244815d690 2023-12-16T21:58:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2448159160 con 0x7f244815d690 2023-12-16T21:58:28.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24480679f0 con 0x7f244815d690 2023-12-16T21:58:28.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f2444ff9640 1 -- 172.21.15.28:0/3333888073 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f244000ab30 con 0x7f244815d690 2023-12-16T21:58:28.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f2444ff9640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2434066470 0x7f2434068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f2444ff9640 1 -- 172.21.15.28:0/3333888073 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f24400839d0 con 0x7f244815d690 2023-12-16T21:58:28.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f2447fff640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2434066470 0x7f2434068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.598+0000 7f2447fff640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2434066470 0x7f2434068930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f24480cd4f0 tx=0x7f2438008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.602+0000 7f2444ff9640 1 -- 172.21.15.28:0/3333888073 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f2440050e30 con 0x7f244815d690 2023-12-16T21:58:28.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24480693d0 con 0x7f244815d690 2023-12-16T21:58:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f2444ff9640 1 -- 172.21.15.28:0/3333888073 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2440018020 con 0x7f244815d690 2023-12-16T21:58:28.764 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-16T21:58:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2434066470 msgr2=0x7f2434068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2434066470 0x7f2434068930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f24480cd4f0 tx=0x7f2438008040 comp rx=0 tx=0).stop 2023-12-16T21:58:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244815d690 msgr2=0x7f244815a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244815d690 0x7f244815a920 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f244000c780 tx=0x7f244000cc40 comp rx=0 tx=0).stop 2023-12-16T21:58:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 shutdown_connections 2023-12-16T21:58:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2434066470 0x7f2434068930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f244815d690 0x7f244815a920 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f244815d2b0 0x7f24481534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 --2- 172.21.15.28:0/3333888073 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f24480d1cf0 0x7f2448152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 >> 172.21.15.28:0/3333888073 conn(0x7f244809c510 msgr2=0x7f24480b4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 shutdown_connections 2023-12-16T21:58:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.762+0000 7f244e6b8640 1 -- 172.21.15.28:0/3333888073 wait complete. 2023-12-16T21:58:28.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:58:28.912 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b407fb640 1 Processor -- start 2023-12-16T21:58:28.912 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b407fb640 1 -- start start 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b407fb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b3809a0e0 0x7f4b380d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b407fb640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b380d1e50 con 0x7f4b380cd270 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b407fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3806c180 con 0x7f4b380d1a40 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b407fb640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3806c460 con 0x7f4b3809a0e0 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b3dd96640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b3809a0e0 0x7f4b380d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b3e597640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4b380cd270 0x7f4b380999d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52006/0 (socket says 172.21.15.28:52006) 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b3dd96640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b3809a0e0 0x7f4b380d0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48670/0 (socket says 172.21.15.28:48670) 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b3e597640 1 -- 172.21.15.28:0/2181073802 learned_addr learned my addr 172.21.15.28:0/2181073802 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b3dd96640 1 -- 172.21.15.28:0/2181073802 >> v1:172.21.15.122:6789/0 conn(0x7f4b380d1a40 legacy=0x7f4b380cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b3dd96640 1 -- 172.21.15.28:0/2181073802 >> v1:172.21.15.28:6789/0 conn(0x7f4b380cd270 legacy=0x7f4b380999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b3d595640 1 -- 172.21.15.28:0/2181073802 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706030419 0 0) 0x7f4b380d1e50 con 0x7f4b380cd270 2023-12-16T21:58:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.910+0000 7f4b3dd96640 1 -- 172.21.15.28:0/2181073802 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b380cf150 con 0x7f4b3809a0e0 2023-12-16T21:58:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b3dd96640 1 --2- 172.21.15.28:0/2181073802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3809a0e0 0x7f4b380d0920 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c0097c0 tx=0x7f4b2c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191d725b54083c5d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b3d595640 1 -- 172.21.15.28:0/2181073802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c0ab650 con 0x7f4b3809a0e0 2023-12-16T21:58:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b3d595640 1 -- 172.21.15.28:0/2181073802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c0ab7b0 con 0x7f4b3809a0e0 2023-12-16T21:58:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b3d595640 1 -- 172.21.15.28:0/2181073802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c0abb40 con 0x7f4b3809a0e0 2023-12-16T21:58:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b407fb640 1 -- 172.21.15.28:0/2181073802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3809a0e0 msgr2=0x7f4b380d0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b407fb640 1 --2- 172.21.15.28:0/2181073802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3809a0e0 0x7f4b380d0920 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c0097c0 tx=0x7f4b2c01c450 comp rx=0 tx=0).stop 2023-12-16T21:58:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b407fb640 1 -- 172.21.15.28:0/2181073802 shutdown_connections 2023-12-16T21:58:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b407fb640 1 --2- 172.21.15.28:0/2181073802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3809a0e0 0x7f4b380d0920 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b407fb640 1 -- 172.21.15.28:0/2181073802 >> 172.21.15.28:0/2181073802 conn(0x7f4b380d0f20 msgr2=0x7f4b380d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b407fb640 1 -- 172.21.15.28:0/2181073802 shutdown_connections 2023-12-16T21:58:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.914+0000 7f4b407fb640 1 -- 172.21.15.28:0/2181073802 wait complete. 2023-12-16T21:58:28.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 Processor -- start 2023-12-16T21:58:28.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 -- start start 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b3809a0e0 0x7f4b38151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b380cd270 0x7f4b380c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380d1a40 0x7f4b380c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b381505d0 con 0x7f4b380d1a40 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b38158280 con 0x7f4b380cd270 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b38158560 con 0x7f4b3809a0e0 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380d1a40 0x7f4b380c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380d1a40 0x7f4b380c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48674/0 (socket says 172.21.15.28:48674) 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 -- 172.21.15.28:0/760694343 learned_addr learned my addr 172.21.15.28:0/760694343 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3e597640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b3809a0e0 0x7f4b38151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3dd96640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b380cd270 0x7f4b380c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 -- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b3809a0e0 msgr2=0x7f4b38151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b3809a0e0 0x7f4b38151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 -- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b380cd270 msgr2=0x7f4b380c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b380cd270 0x7f4b380c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 -- 172.21.15.28:0/760694343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c009420 con 0x7f4b380d1a40 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3dd96640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b380cd270 0x7f4b380c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b3ed98640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380d1a40 0x7f4b380c8530 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800c770 tx=0x7f4b2800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b36ffd640 1 -- 172.21.15.28:0/760694343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28003b80 con 0x7f4b380d1a40 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b36ffd640 1 -- 172.21.15.28:0/760694343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b28003ce0 con 0x7f4b380d1a40 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b36ffd640 1 -- 172.21.15.28:0/760694343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800aa90 con 0x7f4b380d1a40 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b38158840 con 0x7f4b380d1a40 2023-12-16T21:58:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b38150210 con 0x7f4b380d1a40 2023-12-16T21:58:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b380674d0 con 0x7f4b380d1a40 2023-12-16T21:58:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.918+0000 7f4b36ffd640 1 -- 172.21.15.28:0/760694343 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4b2800abf0 con 0x7f4b380d1a40 2023-12-16T21:58:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.922+0000 7f4b36ffd640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b24066470 0x7f4b24068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.922+0000 7f4b36ffd640 1 -- 172.21.15.28:0/760694343 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f4b28083ea0 con 0x7f4b380d1a40 2023-12-16T21:58:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.922+0000 7f4b3e597640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b24066470 0x7f4b24068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.922+0000 7f4b3e597640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b24066470 0x7f4b24068930 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4b30000f70 tx=0x7f4b30009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:28.926+0000 7f4b36ffd640 1 -- 172.21.15.28:0/760694343 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4b28051160 con 0x7f4b380d1a40 2023-12-16T21:58:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4b38158cf0 con 0x7f4b380d1a40 2023-12-16T21:58:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b36ffd640 1 -- 172.21.15.28:0/760694343 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4b28054780 con 0x7f4b380d1a40 2023-12-16T21:58:29.084 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-16T21:58:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b24066470 msgr2=0x7f4b24068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b24066470 0x7f4b24068930 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4b30000f70 tx=0x7f4b30009040 comp rx=0 tx=0).stop 2023-12-16T21:58:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380d1a40 msgr2=0x7f4b380c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.086 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380d1a40 0x7f4b380c8530 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4b2800c770 tx=0x7f4b2800cc30 comp rx=0 tx=0).stop 2023-12-16T21:58:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 shutdown_connections 2023-12-16T21:58:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b24066470 0x7f4b24068930 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380d1a40 0x7f4b380c8530 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b380cd270 0x7f4b380c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 --2- 172.21.15.28:0/760694343 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b3809a0e0 0x7f4b38151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 >> 172.21.15.28:0/760694343 conn(0x7f4b380d0f20 msgr2=0x7f4b380999d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:29.088 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 shutdown_connections 2023-12-16T21:58:29.088 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.082+0000 7f4b407fb640 1 -- 172.21.15.28:0/760694343 wait complete. 2023-12-16T21:58:29.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:58:29.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.218+0000 7f9e25a61640 1 Processor -- start 2023-12-16T21:58:29.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e25a61640 1 -- start start 2023-12-16T21:58:29.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e25a61640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9e200b3c70 0x7f9e200d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e25a61640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200d3880 con 0x7f9e200b35b0 2023-12-16T21:58:29.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e25a61640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200cdc80 con 0x7f9e200cfe90 2023-12-16T21:58:29.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e25a61640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e200d4090 con 0x7f9e200b3c70 2023-12-16T21:58:29.224 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1effd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9e200b35b0 0x7f9e200cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52020/0 (socket says 172.21.15.28:52020) 2023-12-16T21:58:29.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1effd640 1 -- 172.21.15.28:0/3204231779 learned_addr learned my addr 172.21.15.28:0/3204231779 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:29.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1f7fe640 1 --2- 172.21.15.28:0/3204231779 >> v2:172.21.15.122:3300/0 conn(0x7f9e200b3c70 0x7f9e200d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113952768 0 0) 0x7f9e200d3880 con 0x7f9e200b35b0 2023-12-16T21:58:29.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e08003530 con 0x7f9e200b35b0 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108654991 0 0) 0x7f9e08003530 con 0x7f9e200b35b0 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 >> v2:172.21.15.122:3300/0 conn(0x7f9e200b3c70 msgr2=0x7f9e200d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 --2- 172.21.15.28:0/3204231779 >> v2:172.21.15.122:3300/0 conn(0x7f9e200b3c70 0x7f9e200d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 >> v1:172.21.15.122:6789/0 conn(0x7f9e200cfe90 legacy=0x7f9e200ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e200d3ad0 con 0x7f9e200b35b0 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f9e10001a00 con 0x7f9e200b35b0 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 >> v1:172.21.15.28:6789/0 conn(0x7f9e200b35b0 legacy=0x7f9e200cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08003980 0x7f9e08003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e080047d0 0x7f9e08004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08005190 0x7f9e0800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e200d4090 con 0x7f9e08005190 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0800c030 con 0x7f9e08003980 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0800c1f0 con 0x7f9e080047d0 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1f7fe640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08005190 0x7f9e0800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1effd640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08003980 0x7f9e08003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1e7fc640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e080047d0 0x7f9e08004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/3204231779 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e080047d0 msgr2=0x7f9e08004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1f7fe640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e080047d0 0x7f9e08004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/3204231779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08003980 msgr2=0x7f9e08003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1f7fe640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e08003980 0x7f9e08003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.222+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/3204231779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08003530 con 0x7f9e08005190 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/3204231779 reap_dead start 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1f7fe640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08005190 0x7f9e0800b990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009ae0 tx=0x7f9e1801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bef00336064117 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18004020 con 0x7f9e08005190 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e180041c0 con 0x7f9e08005190 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1dffb640 1 -- 172.21.15.28:0/3204231779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180a5680 con 0x7f9e08005190 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- 172.21.15.28:0/3204231779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08005190 msgr2=0x7f9e0800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08005190 0x7f9e0800b990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009ae0 tx=0x7f9e1801ca90 comp rx=0 tx=0).stop 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- 172.21.15.28:0/3204231779 shutdown_connections 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 --2- 172.21.15.28:0/3204231779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08005190 0x7f9e0800b990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- 172.21.15.28:0/3204231779 >> 172.21.15.28:0/3204231779 conn(0x7f9e2009c4b0 msgr2=0x7f9e200d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- 172.21.15.28:0/3204231779 shutdown_connections 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- 172.21.15.28:0/3204231779 wait complete. 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 Processor -- start 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- start start 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e08005190 0x7f9e2014a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e2014aee0 0x7f9e20159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e201493b0 0x7f9e20159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2009d270 con 0x7f9e2014aee0 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e20149900 con 0x7f9e201493b0 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e25a61640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e20149be0 con 0x7f9e08005190 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e2014aee0 0x7f9e20159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e2014aee0 0x7f9e20159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48690/0 (socket says 172.21.15.28:48690) 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1e7fc640 1 -- 172.21.15.28:0/387373829 learned_addr learned my addr 172.21.15.28:0/387373829 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:29.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1effd640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e08005190 0x7f9e2014a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.226+0000 7f9e1f7fe640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e201493b0 0x7f9e20159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1e7fc640 1 -- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e08005190 msgr2=0x7f9e2014a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1e7fc640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e08005190 0x7f9e2014a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1e7fc640 1 -- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e201493b0 msgr2=0x7f9e20159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1e7fc640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e201493b0 0x7f9e20159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1e7fc640 1 -- 172.21.15.28:0/387373829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e20149ec0 con 0x7f9e2014aee0 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1f7fe640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e201493b0 0x7f9e20159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1e7fc640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e2014aee0 0x7f9e20159630 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400e4d0 tx=0x7f9e1400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e07fff640 1 -- 172.21.15.28:0/387373829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e140052e0 con 0x7f9e2014aee0 2023-12-16T21:58:29.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e07fff640 1 -- 172.21.15.28:0/387373829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14005440 con 0x7f9e2014aee0 2023-12-16T21:58:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2014a1a0 con 0x7f9e2014aee0 2023-12-16T21:58:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e07fff640 1 -- 172.21.15.28:0/387373829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14003820 con 0x7f9e2014aee0 2023-12-16T21:58:29.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2014b3e0 con 0x7f9e2014aee0 2023-12-16T21:58:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e20069440 con 0x7f9e2014aee0 2023-12-16T21:58:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e07fff640 1 -- 172.21.15.28:0/387373829 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f9e14003980 con 0x7f9e2014aee0 2023-12-16T21:58:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e07fff640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9e0c066470 0x7f9e0c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e07fff640 1 -- 172.21.15.28:0/387373829 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9e14083c30 con 0x7f9e2014aee0 2023-12-16T21:58:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1effd640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9e0c066470 0x7f9e0c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.230+0000 7f9e1effd640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9e0c066470 0x7f9e0c068930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9e10001340 tx=0x7f9e10009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.234+0000 7f9e07fff640 1 -- 172.21.15.28:0/387373829 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f9e14051020 con 0x7f9e2014aee0 2023-12-16T21:58:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.422+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e200d0720 con 0x7f9e2014aee0 2023-12-16T21:58:29.427 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.422+0000 7f9e07fff640 1 -- 172.21.15.28:0/387373829 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4932 (secure 0 0 0) 0x7f9e14054640 con 0x7f9e2014aee0 2023-12-16T21:58:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:29.427 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:57:42.850798+0000","modified":"2023-12-16T21:57:45.892479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:58:29.427 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2023-12-16T21:58:29.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9e0c066470 msgr2=0x7f9e0c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9e0c066470 0x7f9e0c068930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9e10001340 tx=0x7f9e10009040 comp rx=0 tx=0).stop 2023-12-16T21:58:29.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e2014aee0 msgr2=0x7f9e20159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e2014aee0 0x7f9e20159630 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400e4d0 tx=0x7f9e1400e990 comp rx=0 tx=0).stop 2023-12-16T21:58:29.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 shutdown_connections 2023-12-16T21:58:29.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9e0c066470 0x7f9e0c068930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e201493b0 0x7f9e20159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e2014aee0 0x7f9e20159630 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 --2- 172.21.15.28:0/387373829 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9e08005190 0x7f9e2014a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 >> 172.21.15.28:0/387373829 conn(0x7f9e2009c4b0 msgr2=0x7f9e2006b310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:29.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 shutdown_connections 2023-12-16T21:58:29.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.426+0000 7f9e25a61640 1 -- 172.21.15.28:0/387373829 wait complete. 2023-12-16T21:58:29.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:29.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f9471510640 1 Processor -- start 2023-12-16T21:58:29.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f9471510640 1 -- start start 2023-12-16T21:58:29.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f9471510640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f946c06cdb0 0x7f946c150250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f9471510640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06d190 0x7f946c150790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f9471510640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c070460 con 0x7f946c159da0 2023-12-16T21:58:29.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f9471510640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946c150cd0 con 0x7f946c06cdb0 2023-12-16T21:58:29.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f9471510640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c15a900 con 0x7f946c06d190 2023-12-16T21:58:29.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f94627fc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06d190 0x7f946c150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f946affd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f946c06cdb0 0x7f946c150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f946affd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f946c06cdb0 0x7f946c150250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:58792/0 (socket says 172.21.15.28:58792) 2023-12-16T21:58:29.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f946affd640 1 -- 172.21.15.28:0/2998867926 learned_addr learned my addr 172.21.15.28:0/2998867926 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:29.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f946b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f946c159da0 0x7f946c15a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:42454/0 (socket says 172.21.15.28:42454) 2023-12-16T21:58:29.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f946a7fc640 1 -- 172.21.15.28:0/2998867926 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190352373 0 0) 0x7f946c070460 con 0x7f946c159da0 2023-12-16T21:58:29.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.570+0000 7f946a7fc640 1 -- 172.21.15.28:0/2998867926 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9458003530 con 0x7f946c159da0 2023-12-16T21:58:29.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f94627fc640 1 -- 172.21.15.28:0/2998867926 >> v2:172.21.15.122:3300/0 conn(0x7f946c06cdb0 msgr2=0x7f946c150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:29.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f94627fc640 1 --2- 172.21.15.28:0/2998867926 >> v2:172.21.15.122:3300/0 conn(0x7f946c06cdb0 0x7f946c150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f94627fc640 1 -- 172.21.15.28:0/2998867926 >> v1:172.21.15.122:6789/0 conn(0x7f946c159da0 legacy=0x7f946c15a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f94627fc640 1 -- 172.21.15.28:0/2998867926 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c15ab20 con 0x7f946c06d190 2023-12-16T21:58:29.578 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f94627fc640 1 --2- 172.21.15.28:0/2998867926 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06d190 0x7f946c150790 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f94540097c0 tx=0x7f945401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7610cb4638e1973b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.578 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f946a7fc640 1 -- 172.21.15.28:0/2998867926 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540ab600 con 0x7f946c06d190 2023-12-16T21:58:29.578 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f946a7fc640 1 -- 172.21.15.28:0/2998867926 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94540a7050 con 0x7f946c06d190 2023-12-16T21:58:29.578 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f946a7fc640 1 -- 172.21.15.28:0/2998867926 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540ab960 con 0x7f946c06d190 2023-12-16T21:58:29.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f9471510640 1 -- 172.21.15.28:0/2998867926 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06d190 msgr2=0x7f946c150790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f9471510640 1 --2- 172.21.15.28:0/2998867926 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06d190 0x7f946c150790 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f94540097c0 tx=0x7f945401c3f0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f9471510640 1 -- 172.21.15.28:0/2998867926 shutdown_connections 2023-12-16T21:58:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f9471510640 1 --2- 172.21.15.28:0/2998867926 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06d190 0x7f946c150790 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f9471510640 1 --2- 172.21.15.28:0/2998867926 >> v2:172.21.15.122:3300/0 conn(0x7f946c06cdb0 0x7f946c150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f9471510640 1 -- 172.21.15.28:0/2998867926 >> 172.21.15.28:0/2998867926 conn(0x7f946c06a530 msgr2=0x7f946c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f9471510640 1 -- 172.21.15.28:0/2998867926 shutdown_connections 2023-12-16T21:58:29.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.574+0000 7f9471510640 1 -- 172.21.15.28:0/2998867926 wait complete. 2023-12-16T21:58:29.585 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 Processor -- start 2023-12-16T21:58:29.585 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 -- start start 2023-12-16T21:58:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06cdb0 0x7f946c143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f946c06d190 0x7f946c14bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f946c159da0 0x7f946c0b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c14a9a0 con 0x7f946c159da0 2023-12-16T21:58:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c14ac80 con 0x7f946c06d190 2023-12-16T21:58:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c14af60 con 0x7f946c06cdb0 2023-12-16T21:58:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f946c159da0 0x7f946c0b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f946c159da0 0x7f946c0b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48706/0 (socket says 172.21.15.28:48706) 2023-12-16T21:58:29.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 -- 172.21.15.28:0/4057730143 learned_addr learned my addr 172.21.15.28:0/4057730143 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:29.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f94627fc640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f946c06d190 0x7f946c14bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946affd640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06cdb0 0x7f946c143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 -- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06cdb0 msgr2=0x7f946c143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06cdb0 0x7f946c143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 -- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f946c06d190 msgr2=0x7f946c14bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f946c06d190 0x7f946c14bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 -- 172.21.15.28:0/4057730143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9454009420 con 0x7f946c159da0 2023-12-16T21:58:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f94627fc640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f946c06d190 0x7f946c14bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f946b7fe640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f946c159da0 0x7f946c0b36b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f946400c770 tx=0x7f946400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9463fff640 1 -- 172.21.15.28:0/4057730143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946400ce80 con 0x7f946c159da0 2023-12-16T21:58:29.589 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9463fff640 1 -- 172.21.15.28:0/4057730143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946400a640 con 0x7f946c159da0 2023-12-16T21:58:29.589 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9463fff640 1 -- 172.21.15.28:0/4057730143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946400a9f0 con 0x7f946c159da0 2023-12-16T21:58:29.589 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c14b240 con 0x7f946c159da0 2023-12-16T21:58:29.589 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.578+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c152370 con 0x7f946c159da0 2023-12-16T21:58:29.589 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.582+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c069440 con 0x7f946c159da0 2023-12-16T21:58:29.589 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.582+0000 7f9463fff640 1 -- 172.21.15.28:0/4057730143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f946400ab50 con 0x7f946c159da0 2023-12-16T21:58:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.582+0000 7f9463fff640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f944c066470 0x7f944c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.582+0000 7f9463fff640 1 -- 172.21.15.28:0/4057730143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9464083650 con 0x7f946c159da0 2023-12-16T21:58:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.582+0000 7f946affd640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f944c066470 0x7f944c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.590 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.582+0000 7f946affd640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f944c066470 0x7f944c068930 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f945c009ae0 tx=0x7f945c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.586+0000 7f9463fff640 1 -- 172.21.15.28:0/4057730143 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f9464050af0 con 0x7f946c159da0 2023-12-16T21:58:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.742+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f946c071c10 con 0x7f946c159da0 2023-12-16T21:58:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9463fff640 1 -- 172.21.15.28:0/4057730143 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f9464054110 con 0x7f946c159da0 2023-12-16T21:58:29.748 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:29.748 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:24.764806+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f944c066470 msgr2=0x7f944c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f944c066470 0x7f944c068930 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f945c009ae0 tx=0x7f945c008040 comp rx=0 tx=0).stop 2023-12-16T21:58:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f946c159da0 msgr2=0x7f946c0b36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f946c159da0 0x7f946c0b36b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f946400c770 tx=0x7f946400cc30 comp rx=0 tx=0).stop 2023-12-16T21:58:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 shutdown_connections 2023-12-16T21:58:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f944c066470 0x7f944c068930 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f946c159da0 0x7f946c0b36b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f946c06d190 0x7f946c14bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.751 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 --2- 172.21.15.28:0/4057730143 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f946c06cdb0 0x7f946c143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.751 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 >> 172.21.15.28:0/4057730143 conn(0x7f946c06a530 msgr2=0x7f946c0ce4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:29.751 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 shutdown_connections 2023-12-16T21:58:29.751 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.746+0000 7f9471510640 1 -- 172.21.15.28:0/4057730143 wait complete. 2023-12-16T21:58:29.767 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-16T21:58:29.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:58:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.886+0000 7f9f19c5c640 1 Processor -- start 2023-12-16T21:58:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.886+0000 7f9f19c5c640 1 -- start start 2023-12-16T21:58:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f19c5c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f9f1415c310 0x7f9f140cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f19c5c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f1406b540 0x7f9f1406b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f19c5c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1406bed0 con 0x7f9f141521b0 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f19c5c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f140cc1d0 con 0x7f9f1406b540 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f19c5c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f140cd590 con 0x7f9f1415c310 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f13fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f1406b540 0x7f9f1406b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f13fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f1406b540 0x7f9f1406b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48722/0 (socket says 172.21.15.28:48722) 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f13fff640 1 -- 172.21.15.28:0/3890949426 learned_addr learned my addr 172.21.15.28:0/3890949426 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f12ffd640 1 --2- 172.21.15.28:0/3890949426 >> v2:172.21.15.122:3300/0 conn(0x7f9f1415c310 0x7f9f140cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f13fff640 1 -- 172.21.15.28:0/3890949426 >> v2:172.21.15.122:3300/0 conn(0x7f9f1415c310 msgr2=0x7f9f140cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f13fff640 1 --2- 172.21.15.28:0/3890949426 >> v2:172.21.15.122:3300/0 conn(0x7f9f1415c310 0x7f9f140cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f13fff640 1 -- 172.21.15.28:0/3890949426 >> v1:172.21.15.122:6789/0 conn(0x7f9f141521b0 legacy=0x7f9f140cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f13fff640 1 -- 172.21.15.28:0/3890949426 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f140ccc60 con 0x7f9f1406b540 2023-12-16T21:58:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f13fff640 1 --2- 172.21.15.28:0/3890949426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1406b540 0x7f9f1406b990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9f14066a10 tx=0x7f9f0c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32abb8f20548033a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f127fc640 1 -- 172.21.15.28:0/3890949426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c0ac600 con 0x7f9f1406b540 2023-12-16T21:58:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f127fc640 1 -- 172.21.15.28:0/3890949426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c0a8050 con 0x7f9f1406b540 2023-12-16T21:58:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.890+0000 7f9f127fc640 1 -- 172.21.15.28:0/3890949426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c0ac980 con 0x7f9f1406b540 2023-12-16T21:58:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- 172.21.15.28:0/3890949426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1406b540 msgr2=0x7f9f1406b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/3890949426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1406b540 0x7f9f1406b990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9f14066a10 tx=0x7f9f0c01c490 comp rx=0 tx=0).stop 2023-12-16T21:58:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- 172.21.15.28:0/3890949426 shutdown_connections 2023-12-16T21:58:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/3890949426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1406b540 0x7f9f1406b990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/3890949426 >> v2:172.21.15.122:3300/0 conn(0x7f9f1415c310 0x7f9f140cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- 172.21.15.28:0/3890949426 >> 172.21.15.28:0/3890949426 conn(0x7f9f14070d40 msgr2=0x7f9f14071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- 172.21.15.28:0/3890949426 shutdown_connections 2023-12-16T21:58:29.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- 172.21.15.28:0/3890949426 wait complete. 2023-12-16T21:58:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 Processor -- start 2023-12-16T21:58:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- start start 2023-12-16T21:58:29.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9f1406b540 0x7f9f1414a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f141521b0 0x7f9f141f9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1415c310 0x7f9f14155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f14154cc0 con 0x7f9f141521b0 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1414bb70 con 0x7f9f1415c310 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f19c5c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f141fa390 con 0x7f9f1406b540 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f141521b0 0x7f9f141f9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f141521b0 0x7f9f141f9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48730/0 (socket says 172.21.15.28:48730) 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 -- 172.21.15.28:0/881528368 learned_addr learned my addr 172.21.15.28:0/881528368 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f137fe640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9f1406b540 0x7f9f1414a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f13fff640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1415c310 0x7f9f14155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 -- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9f1406b540 msgr2=0x7f9f1414a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9f1406b540 0x7f9f1414a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 -- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1415c310 msgr2=0x7f9f14155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1415c310 0x7f9f14155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:29.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 -- 172.21.15.28:0/881528368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c0094d0 con 0x7f9f141521b0 2023-12-16T21:58:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f13fff640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1415c310 0x7f9f14155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.894+0000 7f9f12ffd640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f141521b0 0x7f9f141f9d40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00d5f0 tx=0x7f9efc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f0bfff640 1 -- 172.21.15.28:0/881528368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc0125f0 con 0x7f9f141521b0 2023-12-16T21:58:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f0bfff640 1 -- 172.21.15.28:0/881528368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc012750 con 0x7f9f141521b0 2023-12-16T21:58:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f0bfff640 1 -- 172.21.15.28:0/881528368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc012ae0 con 0x7f9f141521b0 2023-12-16T21:58:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f141fa670 con 0x7f9f141521b0 2023-12-16T21:58:29.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f14155a70 con 0x7f9f141521b0 2023-12-16T21:58:29.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f140674d0 con 0x7f9f141521b0 2023-12-16T21:58:29.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f0bfff640 1 -- 172.21.15.28:0/881528368 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f9efc012c40 con 0x7f9f141521b0 2023-12-16T21:58:29.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f0bfff640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9ef4066470 0x7f9ef4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:29.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f0bfff640 1 -- 172.21.15.28:0/881528368 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9efc084400 con 0x7f9f141521b0 2023-12-16T21:58:29.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f137fe640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9ef4066470 0x7f9ef4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:29.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.898+0000 7f9f137fe640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9ef4066470 0x7f9ef4068930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9f04005530 tx=0x7f9f04009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:29.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:29.902+0000 7f9f0bfff640 1 -- 172.21.15.28:0/881528368 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f9efc0517d0 con 0x7f9f141521b0 2023-12-16T21:58:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.090+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f140b10e0 con 0x7f9f141521b0 2023-12-16T21:58:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f0bfff640 1 -- 172.21.15.28:0/881528368 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4932 (secure 0 0 0) 0x7f9efc007380 con 0x7f9f141521b0 2023-12-16T21:58:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:30.096 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:57:42.850798+0000","modified":"2023-12-16T21:57:45.892479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:58:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2023-12-16T21:58:30.098 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9ef4066470 msgr2=0x7f9ef4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.098 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9ef4066470 0x7f9ef4068930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9f04005530 tx=0x7f9f04009040 comp rx=0 tx=0).stop 2023-12-16T21:58:30.098 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f141521b0 msgr2=0x7f9f141f9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f141521b0 0x7f9f141f9d40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00d5f0 tx=0x7f9efc00dab0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 shutdown_connections 2023-12-16T21:58:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f9ef4066470 0x7f9ef4068930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9f1415c310 0x7f9f14155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f141521b0 0x7f9f141f9d40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 --2- 172.21.15.28:0/881528368 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f9f1406b540 0x7f9f1414a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 >> 172.21.15.28:0/881528368 conn(0x7f9f14070d40 msgr2=0x7f9f1406e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 shutdown_connections 2023-12-16T21:58:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.094+0000 7f9f19c5c640 1 -- 172.21.15.28:0/881528368 wait complete. 2023-12-16T21:58:30.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:58:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.222+0000 7fb61dc01640 1 Processor -- start 2023-12-16T21:58:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.222+0000 7fb61dc01640 1 -- start start 2023-12-16T21:58:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb61dc01640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fb6180cfe90 0x7fb6180d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb61dc01640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb6180b3c70 0x7fb61815f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb61dc01640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6180d3880 con 0x7fb6180b35b0 2023-12-16T21:58:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb61dc01640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6180d4090 con 0x7fb6180cfe90 2023-12-16T21:58:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb61dc01640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180cdc80 con 0x7fb6180b3c70 2023-12-16T21:58:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6177fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb6180b35b0 0x7fb6180cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52030/0 (socket says 172.21.15.28:52030) 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6177fe640 1 -- 172.21.15.28:0/698696570 learned_addr learned my addr 172.21.15.28:0/698696570 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827818297 0 0) 0x7fb6180d3880 con 0x7fb6180b35b0 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb600003530 con 0x7fb6180b35b0 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb617fff640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb6180b3c70 0x7fb61815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb616ffd640 1 --2- 172.21.15.28:0/698696570 >> v2:172.21.15.122:3300/0 conn(0x7fb6180cfe90 0x7fb6180d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3798784329 0 0) 0x7fb600003530 con 0x7fb6180b35b0 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb6180b3c70 msgr2=0x7fb61815f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb6180b3c70 0x7fb61815f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 >> v2:172.21.15.122:3300/0 conn(0x7fb6180cfe90 msgr2=0x7fb6180d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 --2- 172.21.15.28:0/698696570 >> v2:172.21.15.122:3300/0 conn(0x7fb6180cfe90 0x7fb6180d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6180d3ad0 con 0x7fb6180b35b0 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fb608001680 con 0x7fb6180b35b0 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 >> v1:172.21.15.28:6789/0 conn(0x7fb6180b35b0 legacy=0x7fb6180cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb600003980 0x7fb600003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb6180b3c70 0x7fb6000047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb600004d10 0x7fb600005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180d4090 con 0x7fb600003980 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180cdc80 con 0x7fb600004d10 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60000b990 con 0x7fb6180b3c70 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6177fe640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb600003980 0x7fb600003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb616ffd640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb6180b3c70 0x7fb6000047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb617fff640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb600004d10 0x7fb600005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6177fe640 1 -- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb6180b3c70 msgr2=0x7fb6000047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6177fe640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb6180b3c70 0x7fb6000047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6177fe640 1 -- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb600004d10 msgr2=0x7fb600005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6177fe640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb600004d10 0x7fb600005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.226+0000 7fb6177fe640 1 -- 172.21.15.28:0/698696570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb600003530 con 0x7fb600003980 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb6177fe640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb600003980 0x7fb600003da0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb60800c290 tx=0x7fb60801ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14ae338b1eeee97 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6080a5900 con 0x7fb600003980 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6080a5a60 con 0x7fb600003980 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb6167fc640 1 -- 172.21.15.28:0/698696570 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6080a5e10 con 0x7fb600003980 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- 172.21.15.28:0/698696570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb600003980 msgr2=0x7fb600003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 --2- 172.21.15.28:0/698696570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb600003980 0x7fb600003da0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb60800c290 tx=0x7fb60801ccb0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 -- 172.21.15.28:0/698696570 reap_dead start 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- 172.21.15.28:0/698696570 shutdown_connections 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- 172.21.15.28:0/698696570 >> 172.21.15.28:0/698696570 conn(0x7fb61809c4b0 msgr2=0x7fb6180d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- 172.21.15.28:0/698696570 shutdown_connections 2023-12-16T21:58:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- 172.21.15.28:0/698696570 wait complete. 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 Processor -- start 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- start start 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb600003980 0x7fb6180ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb618158f80 0x7fb6181f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6181586a0 0x7fb61815d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180cb4a0 con 0x7fb6181586a0 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6181f68e0 con 0x7fb618158f80 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb61dc01640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6180ca820 con 0x7fb600003980 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6181586a0 0x7fb61815d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6181586a0 0x7fb61815d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48754/0 (socket says 172.21.15.28:48754) 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 -- 172.21.15.28:0/291648177 learned_addr learned my addr 172.21.15.28:0/291648177 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb6177fe640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb600003980 0x7fb6180ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb616ffd640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb618158f80 0x7fb6181f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 -- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb600003980 msgr2=0x7fb6180ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb600003980 0x7fb6180ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 -- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb618158f80 msgr2=0x7fb6181f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb618158f80 0x7fb6181f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 -- 172.21.15.28:0/291648177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6180cab00 con 0x7fb6181586a0 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb616ffd640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb618158f80 0x7fb6181f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.230+0000 7fb617fff640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6181586a0 0x7fb61815d210 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb6180ca1d0 tx=0x7fb6100115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb5fffff640 1 -- 172.21.15.28:0/291648177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61000e070 con 0x7fb6181586a0 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb5fffff640 1 -- 172.21.15.28:0/291648177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb610011e10 con 0x7fb6181586a0 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6180cade0 con 0x7fb6181586a0 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb5fffff640 1 -- 172.21.15.28:0/291648177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6100039a0 con 0x7fb6181586a0 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb618159680 con 0x7fb6181586a0 2023-12-16T21:58:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6180cb5e0 con 0x7fb6181586a0 2023-12-16T21:58:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb5fffff640 1 -- 172.21.15.28:0/291648177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fb610003b00 con 0x7fb6181586a0 2023-12-16T21:58:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb5fffff640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb604066470 0x7fb604068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb5fffff640 1 -- 172.21.15.28:0/291648177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fb610084450 con 0x7fb6181586a0 2023-12-16T21:58:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb6177fe640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb604066470 0x7fb604068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.234+0000 7fb6177fe640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb604066470 0x7fb604068930 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb6080aa000 tx=0x7fb608001a10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.238+0000 7fb5fffff640 1 -- 172.21.15.28:0/291648177 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fb6180cb5e0 con 0x7fb6181586a0 2023-12-16T21:58:30.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6180cb5e0 con 0x7fb6181586a0 2023-12-16T21:58:30.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb5fffff640 1 -- 172.21.15.28:0/291648177 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4932 (secure 0 0 0) 0x7fb6180cb5e0 con 0x7fb6181586a0 2023-12-16T21:58:30.429 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:30.429 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1678879150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1678879150},{"type":"v1","addr":"172.21.15.28:6835","nonce":1678879150}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3867677976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3867677976},{"type":"v1","addr":"172.21.15.122:6837","nonce":3867677976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/966572348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":966572348},{"type":"v1","addr":"172.21.15.28:6834","nonce":966572348}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:57:42.850798+0000","modified":"2023-12-16T21:57:45.892479+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6835/771354521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":771354521},{"type":"v1","addr":"172.21.15.122:6835","nonce":771354521}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:58:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2023-12-16T21:58:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb604066470 msgr2=0x7fb604068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb604066470 0x7fb604068930 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb6080aa000 tx=0x7fb608001a10 comp rx=0 tx=0).stop 2023-12-16T21:58:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6181586a0 msgr2=0x7fb61815d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6181586a0 0x7fb61815d210 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb6180ca1d0 tx=0x7fb6100115a0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 shutdown_connections 2023-12-16T21:58:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb604066470 0x7fb604068930 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6181586a0 0x7fb61815d210 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb618158f80 0x7fb6181f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 --2- 172.21.15.28:0/291648177 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb600003980 0x7fb6180ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 >> 172.21.15.28:0/291648177 conn(0x7fb61809c4b0 msgr2=0x7fb6180b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 shutdown_connections 2023-12-16T21:58:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.426+0000 7fb61dc01640 1 -- 172.21.15.28:0/291648177 wait complete. 2023-12-16T21:58:30.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.558+0000 7fde88794640 1 Processor -- start 2023-12-16T21:58:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.558+0000 7fde88794640 1 -- start start 2023-12-16T21:58:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde88794640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fde80144840 0x7fde8006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde88794640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80072180 0x7fde8006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde88794640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800710e0 con 0x7fde80070460 2023-12-16T21:58:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde88794640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8015f010 con 0x7fde80144840 2023-12-16T21:58:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde88794640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800713c0 con 0x7fde80072180 2023-12-16T21:58:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fde80070460 0x7fde800708e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52038/0 (socket says 172.21.15.28:52038) 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 -- 172.21.15.28:0/1903266920 learned_addr learned my addr 172.21.15.28:0/1903266920 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde85d2f640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80072180 0x7fde8006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86530640 1 --2- 172.21.15.28:0/1903266920 >> v2:172.21.15.122:3300/0 conn(0x7fde80144840 0x7fde8006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667916931 0 0) 0x7fde800710e0 con 0x7fde80070460 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde6c003530 con 0x7fde80070460 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 362890991 0 0) 0x7fde6c003530 con 0x7fde80070460 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80072180 msgr2=0x7fde8006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80072180 0x7fde8006c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 >> v2:172.21.15.122:3300/0 conn(0x7fde80144840 msgr2=0x7fde8006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 --2- 172.21.15.28:0/1903266920 >> v2:172.21.15.122:3300/0 conn(0x7fde80144840 0x7fde8006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800716a0 con 0x7fde80070460 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fde78001a00 con 0x7fde80070460 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 >> v1:172.21.15.28:6789/0 conn(0x7fde80070460 legacy=0x7fde800708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80072180 0x7fde6c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde6c003f00 0x7fde6c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c004d10 0x7fde6c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800713c0 con 0x7fde6c004d10 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c00b990 con 0x7fde6c003f00 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c00bad0 con 0x7fde80072180 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c004d10 0x7fde6c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86530640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80072180 0x7fde6c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde85d2f640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde6c003f00 0x7fde6c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 -- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80072180 msgr2=0x7fde6c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80072180 0x7fde6c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 -- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde6c003f00 msgr2=0x7fde6c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde6c003f00 0x7fde6c0047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 -- 172.21.15.28:0/1903266920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c003530 con 0x7fde6c004d10 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde86d31640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c004d10 0x7fde6c005190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fde7800a830 tx=0x7fde7801d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6679869acf03a984 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde780ad630 con 0x7fde6c004d10 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde780a8070 con 0x7fde6c004d10 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.562+0000 7fde8552e640 1 -- 172.21.15.28:0/1903266920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde780ada30 con 0x7fde6c004d10 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- 172.21.15.28:0/1903266920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c004d10 msgr2=0x7fde6c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 --2- 172.21.15.28:0/1903266920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6c004d10 0x7fde6c005190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fde7800a830 tx=0x7fde7801d2e0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86d31640 1 -- 172.21.15.28:0/1903266920 reap_dead start 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- 172.21.15.28:0/1903266920 shutdown_connections 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- 172.21.15.28:0/1903266920 >> 172.21.15.28:0/1903266920 conn(0x7fde8006a530 msgr2=0x7fde8006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- 172.21.15.28:0/1903266920 shutdown_connections 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- 172.21.15.28:0/1903266920 wait complete. 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 Processor -- start 2023-12-16T21:58:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- start start 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde800c7010 0x7fde801f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde800c73f0 0x7fde801335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80133b30 0x7fde800b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b2e70 con 0x7fde800c7010 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b21f0 con 0x7fde800c73f0 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b24d0 con 0x7fde80133b30 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde800c7010 0x7fde801f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde800c7010 0x7fde801f37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48770/0 (socket says 172.21.15.28:48770) 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 -- 172.21.15.28:0/4242716233 learned_addr learned my addr 172.21.15.28:0/4242716233 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86d31640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80133b30 0x7fde800b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde85d2f640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde800c73f0 0x7fde801335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 -- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80133b30 msgr2=0x7fde800b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80133b30 0x7fde800b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 -- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde800c73f0 msgr2=0x7fde801335f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde800c73f0 0x7fde801335f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 -- 172.21.15.28:0/4242716233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800b27b0 con 0x7fde800c7010 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde85d2f640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde800c73f0 0x7fde801335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde86530640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde800c7010 0x7fde801f37a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fde7c011660 tx=0x7fde7c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde76ffd640 1 -- 172.21.15.28:0/4242716233 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c00f9c0 con 0x7fde800c7010 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde76ffd640 1 -- 172.21.15.28:0/4242716233 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c00fb20 con 0x7fde800c7010 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800b2a90 con 0x7fde800c7010 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde76ffd640 1 -- 172.21.15.28:0/4242716233 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c00fe90 con 0x7fde800c7010 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde801363a0 con 0x7fde800c7010 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.566+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8006c780 con 0x7fde800c7010 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.570+0000 7fde76ffd640 1 -- 172.21.15.28:0/4242716233 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fde7c015460 con 0x7fde800c7010 2023-12-16T21:58:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.570+0000 7fde76ffd640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fde64066470 0x7fde64068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.570+0000 7fde76ffd640 1 -- 172.21.15.28:0/4242716233 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fde7c084760 con 0x7fde800c7010 2023-12-16T21:58:30.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.570+0000 7fde85d2f640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fde64066470 0x7fde64068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.570+0000 7fde85d2f640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fde64066470 0x7fde64068930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde800c6d80 tx=0x7fde70008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.574+0000 7fde76ffd640 1 -- 172.21.15.28:0/4242716233 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fde7c051cc0 con 0x7fde800c7010 2023-12-16T21:58:30.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.722+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde80069440 con 0x7fde800c7010 2023-12-16T21:58:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde76ffd640 1 -- 172.21.15.28:0/4242716233 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+16789 (secure 0 0 0) 0x7fde7c0552e0 con 0x7fde800c7010 2023-12-16T21:58:30.728 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:30.728 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:24.764806+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:57:41.198086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:57:41.842692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":3,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fde64066470 msgr2=0x7fde64068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fde64066470 0x7fde64068930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde800c6d80 tx=0x7fde70008040 comp rx=0 tx=0).stop 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde800c7010 msgr2=0x7fde801f37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde800c7010 0x7fde801f37a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fde7c011660 tx=0x7fde7c011b20 comp rx=0 tx=0).stop 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 shutdown_connections 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fde64066470 0x7fde64068930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fde80133b30 0x7fde800b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fde800c73f0 0x7fde801335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 --2- 172.21.15.28:0/4242716233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde800c7010 0x7fde801f37a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 >> 172.21.15.28:0/4242716233 conn(0x7fde8006a530 msgr2=0x7fde801428e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 shutdown_connections 2023-12-16T21:58:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.726+0000 7fde88794640 1 -- 172.21.15.28:0/4242716233 wait complete. 2023-12-16T21:58:30.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-16T21:58:30.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f182d98a640 1 Processor -- start 2023-12-16T21:58:30.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f182d98a640 1 -- start start 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f182d98a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f182806cdb0 0x7f1828150250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f182d98a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f182806d190 0x7f1828150790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f182d98a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1828159da0 0x7f182815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f182d98a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1828150cd0 con 0x7f1828159da0 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f182d98a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1828070460 con 0x7f182806d190 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f182d98a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182815ab20 con 0x7f182806cdb0 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18277fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1828159da0 0x7f182815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18277fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1828159da0 0x7f182815a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48774/0 (socket says 172.21.15.28:48774) 2023-12-16T21:58:30.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18277fe640 1 -- 172.21.15.28:0/3759126104 learned_addr learned my addr 172.21.15.28:0/3759126104 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f1826ffd640 1 --2- 172.21.15.28:0/3759126104 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f182806cdb0 0x7f1828150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18267fc640 1 --2- 172.21.15.28:0/3759126104 >> v2:172.21.15.122:3300/0 conn(0x7f182806d190 0x7f1828150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18277fe640 1 -- 172.21.15.28:0/3759126104 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f182806cdb0 msgr2=0x7f1828150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18277fe640 1 --2- 172.21.15.28:0/3759126104 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f182806cdb0 0x7f1828150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18277fe640 1 -- 172.21.15.28:0/3759126104 >> v2:172.21.15.122:3300/0 conn(0x7f182806d190 msgr2=0x7f1828150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18277fe640 1 --2- 172.21.15.28:0/3759126104 >> v2:172.21.15.122:3300/0 conn(0x7f182806d190 0x7f1828150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.854+0000 7f18277fe640 1 -- 172.21.15.28:0/3759126104 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828070690 con 0x7f1828159da0 2023-12-16T21:58:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f18277fe640 1 --2- 172.21.15.28:0/3759126104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828159da0 0x7f182815a1f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1828067f30 tx=0x7f181801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1824eef660f5912c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f1825ffb640 1 -- 172.21.15.28:0/3759126104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18180a5630 con 0x7f1828159da0 2023-12-16T21:58:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f1825ffb640 1 -- 172.21.15.28:0/3759126104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18180a5790 con 0x7f1828159da0 2023-12-16T21:58:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f1825ffb640 1 -- 172.21.15.28:0/3759126104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18180a5b20 con 0x7f1828159da0 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 -- 172.21.15.28:0/3759126104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828159da0 msgr2=0x7f182815a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 --2- 172.21.15.28:0/3759126104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828159da0 0x7f182815a1f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1828067f30 tx=0x7f181801c940 comp rx=0 tx=0).stop 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 -- 172.21.15.28:0/3759126104 shutdown_connections 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 --2- 172.21.15.28:0/3759126104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1828159da0 0x7f182815a1f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 --2- 172.21.15.28:0/3759126104 >> v2:172.21.15.122:3300/0 conn(0x7f182806d190 0x7f1828150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 --2- 172.21.15.28:0/3759126104 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f182806cdb0 0x7f1828150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 -- 172.21.15.28:0/3759126104 >> 172.21.15.28:0/3759126104 conn(0x7f182806a530 msgr2=0x7f182806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 -- 172.21.15.28:0/3759126104 shutdown_connections 2023-12-16T21:58:30.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 -- 172.21.15.28:0/3759126104 wait complete. 2023-12-16T21:58:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 Processor -- start 2023-12-16T21:58:30.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.858+0000 7f182d98a640 1 -- start start 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f182806cdb0 0x7f182812f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f182806d190 0x7f182814b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1828159da0 0x7f18280af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18280affb0 con 0x7f182806cdb0 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18280b0290 con 0x7f182806d190 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18280b0570 con 0x7f1828159da0 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f182806cdb0 0x7f182812f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f182806cdb0 0x7f182812f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48776/0 (socket says 172.21.15.28:48776) 2023-12-16T21:58:30.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 -- 172.21.15.28:0/3640935543 learned_addr learned my addr 172.21.15.28:0/3640935543 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f18277fe640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1828159da0 0x7f18280af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 -- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1828159da0 msgr2=0x7f18280af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f18267fc640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f182806d190 0x7f182814b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1828159da0 0x7f18280af2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 -- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f182806d190 msgr2=0x7f182814b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f182806d190 0x7f182814b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 -- 172.21.15.28:0/3640935543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818009b30 con 0x7f182806cdb0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f1826ffd640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f182806cdb0 0x7f182812f740 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f182000d800 tx=0x7f182000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f181ffff640 1 -- 172.21.15.28:0/3640935543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18200096a0 con 0x7f182806cdb0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f181ffff640 1 -- 172.21.15.28:0/3640935543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1820009800 con 0x7f182806cdb0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18281439f0 con 0x7f182806cdb0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f181ffff640 1 -- 172.21.15.28:0/3640935543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1820009b70 con 0x7f182806cdb0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18280af7f0 con 0x7f182806cdb0 2023-12-16T21:58:30.866 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1828143850 con 0x7f182806cdb0 2023-12-16T21:58:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f181ffff640 1 -- 172.21.15.28:0/3640935543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1820009cd0 con 0x7f182806cdb0 2023-12-16T21:58:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f181ffff640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18080664a0 0x7f1808068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f181ffff640 1 -- 172.21.15.28:0/3640935543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f1820084690 con 0x7f182806cdb0 2023-12-16T21:58:30.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.862+0000 7f18267fc640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18080664a0 0x7f1808068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:30.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.866+0000 7f18267fc640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18080664a0 0x7f1808068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1828149d50 tx=0x7f1810008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:30.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:30.866+0000 7f181ffff640 1 -- 172.21.15.28:0/3640935543 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f1828143850 con 0x7f182806cdb0 2023-12-16T21:58:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.050+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1828143850 con 0x7f182806cdb0 2023-12-16T21:58:31.056 INFO:tasks.ceph.mon.a.smithi028.stderr:2023-12-16T21:58:31.054+0000 7fbb08a8a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T21:58:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.054+0000 7f181ffff640 1 -- 172.21.15.28:0/3640935543 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1828143850 con 0x7f182806cdb0 2023-12-16T21:58:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-16T21:58:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18080664a0 msgr2=0x7f1808068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18080664a0 0x7f1808068960 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1828149d50 tx=0x7f1810008040 comp rx=0 tx=0).stop 2023-12-16T21:58:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f182806cdb0 msgr2=0x7f182812f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f182806cdb0 0x7f182812f740 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f182000d800 tx=0x7f182000dcc0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 shutdown_connections 2023-12-16T21:58:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18080664a0 0x7f1808068960 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1828159da0 0x7f18280af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f182806d190 0x7f182814b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 --2- 172.21.15.28:0/3640935543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f182806cdb0 0x7f182812f740 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 >> 172.21.15.28:0/3640935543 conn(0x7f182806a530 msgr2=0x7f182814e200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 shutdown_connections 2023-12-16T21:58:31.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.058+0000 7f182d98a640 1 -- 172.21.15.28:0/3640935543 wait complete. 2023-12-16T21:58:31.073 INFO:tasks.ceph.mds.d.smithi122.stderr: -2954> 2023-12-16T21:57:47.695+0000 7f3d98f9f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:58:31.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-16T21:58:31.126 INFO:tasks.ceph.mds.d.smithi122.stdout:starting mds.d at 2023-12-16T21:58:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.186+0000 7fc569ef9640 1 Processor -- start 2023-12-16T21:58:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.186+0000 7fc569ef9640 1 -- start start 2023-12-16T21:58:31.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.186+0000 7fc569ef9640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc56415adb0 0x7fc56413d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:31.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.186+0000 7fc569ef9640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56413d2b0 con 0x7fc56413f5f0 2023-12-16T21:58:31.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.186+0000 7fc569ef9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640cdd70 con 0x7fc564071d90 2023-12-16T21:58:31.194 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.186+0000 7fc569ef9640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc564140520 con 0x7fc56415adb0 2023-12-16T21:58:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.186+0000 7fc563fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc56413f5f0 0x7fc5640ce200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52048/0 (socket says 172.21.15.28:52048) 2023-12-16T21:58:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.186+0000 7fc563fff640 1 -- 172.21.15.28:0/1199877121 learned_addr learned my addr 172.21.15.28:0/1199877121 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc562ffd640 1 --2- 172.21.15.28:0/1199877121 >> v2:172.21.15.122:3300/0 conn(0x7fc56415adb0 0x7fc56413d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695268874 0 0) 0x7fc56413d2b0 con 0x7fc56413f5f0 2023-12-16T21:58:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc54c003530 con 0x7fc56413f5f0 2023-12-16T21:58:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1748769355 0 0) 0x7fc54c003530 con 0x7fc56413f5f0 2023-12-16T21:58:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 >> v2:172.21.15.122:3300/0 conn(0x7fc56415adb0 msgr2=0x7fc56413d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 --2- 172.21.15.28:0/1199877121 >> v2:172.21.15.122:3300/0 conn(0x7fc56415adb0 0x7fc56413d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 >> v1:172.21.15.122:6789/0 conn(0x7fc564071d90 legacy=0x7fc564072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56413ff60 con 0x7fc56413f5f0 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fc55c001680 con 0x7fc56413f5f0 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 >> v1:172.21.15.28:6789/0 conn(0x7fc56413f5f0 legacy=0x7fc5640ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc54c0039d0 0x7fc54c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54c0047d0 0x7fc54c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c005190 0x7fc54c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc564140520 con 0x7fc54c0047d0 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c00c090 con 0x7fc54c005190 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c00c250 con 0x7fc54c0039d0 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc562ffd640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54c0047d0 0x7fc54c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc563fff640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c005190 0x7fc54c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5637fe640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc54c0039d0 0x7fc54c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:31.196 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc562ffd640 1 -- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc54c0039d0 msgr2=0x7fc54c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc562ffd640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc54c0039d0 0x7fc54c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc562ffd640 1 -- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c005190 msgr2=0x7fc54c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc562ffd640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c005190 0x7fc54c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc562ffd640 1 -- 172.21.15.28:0/1199877121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c003530 con 0x7fc54c0047d0 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc563fff640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c005190 0x7fc54c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc563fff640 1 -- 172.21.15.28:0/1199877121 reap_dead start 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc562ffd640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54c0047d0 0x7fc54c004c50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc558009ae0 tx=0x7fc55801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db94b6cf128c7835 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580a5530 con 0x7fc54c0047d0 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5580a5690 con 0x7fc54c0047d0 2023-12-16T21:58:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc5627fc640 1 -- 172.21.15.28:0/1199877121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5580a5a40 con 0x7fc54c0047d0 2023-12-16T21:58:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc569ef9640 1 -- 172.21.15.28:0/1199877121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54c0047d0 msgr2=0x7fc54c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc569ef9640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54c0047d0 0x7fc54c004c50 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc558009ae0 tx=0x7fc55801ca50 comp rx=0 tx=0).stop 2023-12-16T21:58:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc569ef9640 1 -- 172.21.15.28:0/1199877121 shutdown_connections 2023-12-16T21:58:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc569ef9640 1 --2- 172.21.15.28:0/1199877121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc54c0047d0 0x7fc54c004c50 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc569ef9640 1 -- 172.21.15.28:0/1199877121 >> 172.21.15.28:0/1199877121 conn(0x7fc5640b0cc0 msgr2=0x7fc5640b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc569ef9640 1 -- 172.21.15.28:0/1199877121 shutdown_connections 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.190+0000 7fc569ef9640 1 -- 172.21.15.28:0/1199877121 wait complete. 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 Processor -- start 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 -- start start 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c0047d0 0x7fc56413c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc5640a9df0 0x7fc5640aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5640aa7b0 0x7fc5640b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640aded0 con 0x7fc5640aa7b0 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640ae2f0 con 0x7fc54c0047d0 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5640aac10 con 0x7fc5640a9df0 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5640aa7b0 0x7fc5640b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5640aa7b0 0x7fc5640b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48784/0 (socket says 172.21.15.28:48784) 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 -- 172.21.15.28:0/3929094619 learned_addr learned my addr 172.21.15.28:0/3929094619 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc562ffd640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc5640a9df0 0x7fc5640aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc5637fe640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c0047d0 0x7fc56413c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 -- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc5640a9df0 msgr2=0x7fc5640aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc5640a9df0 0x7fc5640aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 -- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c0047d0 msgr2=0x7fc56413c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c0047d0 0x7fc56413c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 -- 172.21.15.28:0/3929094619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5640ad250 con 0x7fc5640aa7b0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc563fff640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5640aa7b0 0x7fc5640b2dd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc564143bd0 tx=0x7fc55c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc54bfff640 1 -- 172.21.15.28:0/3929094619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00f840 con 0x7fc5640aa7b0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc54bfff640 1 -- 172.21.15.28:0/3929094619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c00f9e0 con 0x7fc5640aa7b0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5640ad530 con 0x7fc5640aa7b0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc54bfff640 1 -- 172.21.15.28:0/3929094619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00fd90 con 0x7fc5640aa7b0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5640adaf0 con 0x7fc5640aa7b0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.194+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5640693d0 con 0x7fc5640aa7b0 2023-12-16T21:58:31.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.198+0000 7fc54bfff640 1 -- 172.21.15.28:0/3929094619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fc55c01f460 con 0x7fc5640aa7b0 2023-12-16T21:58:31.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.198+0000 7fc54bfff640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc550066460 0x7fc550068920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:31.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.198+0000 7fc54bfff640 1 -- 172.21.15.28:0/3929094619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fc55c086d30 con 0x7fc5640aa7b0 2023-12-16T21:58:31.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.198+0000 7fc5637fe640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc550066460 0x7fc550068920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:31.203 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.198+0000 7fc5637fe640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc550066460 0x7fc550068920 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc5640b4ca0 tx=0x7fc554009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:31.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.202+0000 7fc54bfff640 1 -- 172.21.15.28:0/3929094619 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fc55c054150 con 0x7fc5640aa7b0 2023-12-16T21:58:31.391 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:31.386+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc56406e710 con 0x7fc5640aa7b0 2023-12-16T21:58:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.066+0000 7fc54bfff640 1 -- 172.21.15.28:0/3929094619 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc55c011ea0 con 0x7fc5640aa7b0 2023-12-16T21:58:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc550066460 msgr2=0x7fc550068920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc550066460 0x7fc550068920 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc5640b4ca0 tx=0x7fc554009040 comp rx=0 tx=0).stop 2023-12-16T21:58:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5640aa7b0 msgr2=0x7fc5640b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5640aa7b0 0x7fc5640b2dd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc564143bd0 tx=0x7fc55c011b60 comp rx=0 tx=0).stop 2023-12-16T21:58:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 shutdown_connections 2023-12-16T21:58:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc550066460 0x7fc550068920 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5640aa7b0 0x7fc5640b2dd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc5640a9df0 0x7fc5640aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 --2- 172.21.15.28:0/3929094619 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc54c0047d0 0x7fc56413c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 >> 172.21.15.28:0/3929094619 conn(0x7fc5640b0cc0 msgr2=0x7fc5640afb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 shutdown_connections 2023-12-16T21:58:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.070+0000 7fc569ef9640 1 -- 172.21.15.28:0/3929094619 wait complete. 2023-12-16T21:58:32.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-16T21:58:32.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d45cc8640 1 Processor -- start 2023-12-16T21:58:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d45cc8640 1 -- start start 2023-12-16T21:58:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d45cc8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 0x7f8d400ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d45cc8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400ce370 con 0x7f8d400ae620 2023-12-16T21:58:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d45cc8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400cd760 con 0x7f8d4015a480 2023-12-16T21:58:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d45cc8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400cd570 con 0x7f8d4006de00 2023-12-16T21:58:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d37fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8d400ae620 0x7f8d4006d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52056/0 (socket says 172.21.15.28:52056) 2023-12-16T21:58:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d37fff640 1 -- 172.21.15.28:0/1266271896 learned_addr learned my addr 172.21.15.28:0/1266271896 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.226+0000 7f8d3ffff640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 0x7f8d400ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545781012 0 0) 0x7f8d400ce370 con 0x7f8d400ae620 2023-12-16T21:58:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d2c005740 con 0x7f8d400ae620 2023-12-16T21:58:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564468365 0 0) 0x7f8d400cd760 con 0x7f8d4015a480 2023-12-16T21:58:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d400ce370 con 0x7f8d4015a480 2023-12-16T21:58:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1708039787 0 0) 0x7f8d2c005740 con 0x7f8d400ae620 2023-12-16T21:58:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 msgr2=0x7f8d400ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 0x7f8d400ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 >> v1:172.21.15.122:6789/0 conn(0x7f8d4015a480 legacy=0x7f8d400cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400cda40 con 0x7f8d400ae620 2023-12-16T21:58:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f8d28001600 con 0x7f8d400ae620 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 >> v1:172.21.15.28:6789/0 conn(0x7f8d400ae620 legacy=0x7f8d4006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d2c005d50 0x7f8d2c006150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d2c006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 0x7f8d2c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c00dc50 con 0x7f8d2c0069e0 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c00ddc0 con 0x7f8d2c005d50 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c00df80 con 0x7f8d4006de00 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d37fff640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d2c006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3ffff640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 0x7f8d2c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3f7fe640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d2c005d50 0x7f8d2c006150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d37fff640 1 -- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 msgr2=0x7f8d2c0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d37fff640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 0x7f8d2c0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d37fff640 1 -- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d2c005d50 msgr2=0x7f8d2c006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d37fff640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d2c005d50 0x7f8d2c006150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d37fff640 1 -- 172.21.15.28:0/1266271896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c005740 con 0x7f8d2c0069e0 2023-12-16T21:58:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3ffff640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d4006de00 0x7f8d2c0073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d37fff640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d2c006e60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d28007f20 tx=0x7f8d2801ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b299a6c140c1fcd3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d28005ba0 con 0x7f8d2c0069e0 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d28005d20 con 0x7f8d2c0069e0 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3effd640 1 -- 172.21.15.28:0/1266271896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d280072b0 con 0x7f8d2c0069e0 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d45cc8640 1 -- 172.21.15.28:0/1266271896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 msgr2=0x7f8d2c006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d45cc8640 1 --2- 172.21.15.28:0/1266271896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d2c006e60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d28007f20 tx=0x7f8d2801ca80 comp rx=0 tx=0).stop 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d3ffff640 1 -- 172.21.15.28:0/1266271896 reap_dead start 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d45cc8640 1 -- 172.21.15.28:0/1266271896 shutdown_connections 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.230+0000 7f8d45cc8640 1 -- 172.21.15.28:0/1266271896 >> 172.21.15.28:0/1266271896 conn(0x7f8d400b0c50 msgr2=0x7f8d400b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- 172.21.15.28:0/1266271896 shutdown_connections 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- 172.21.15.28:0/1266271896 wait complete. 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 Processor -- start 2023-12-16T21:58:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- start start 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d401528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d400725b0 0x7f8d400b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d400b20c0 0x7f8d40070930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400b2540 con 0x7f8d2c0069e0 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400b29e0 con 0x7f8d400725b0 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40071510 con 0x7f8d400b20c0 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d401528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d401528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48804/0 (socket says 172.21.15.28:48804) 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 -- 172.21.15.28:0/302207694 learned_addr learned my addr 172.21.15.28:0/302207694 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d37fff640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d400725b0 0x7f8d400b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3ffff640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d400b20c0 0x7f8d40070930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 -- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d400b20c0 msgr2=0x7f8d40070930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d400b20c0 0x7f8d40070930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 -- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d400725b0 msgr2=0x7f8d400b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d400725b0 0x7f8d400b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 -- 172.21.15.28:0/302207694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400717f0 con 0x7f8d2c0069e0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3ffff640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d400b20c0 0x7f8d40070930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3f7fe640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d401528e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000f8c0 tx=0x7f8d3000fd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3cff9640 1 -- 172.21.15.28:0/302207694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3000d8d0 con 0x7f8d2c0069e0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3cff9640 1 -- 172.21.15.28:0/302207694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3000da50 con 0x7f8d2c0069e0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d40071ad0 con 0x7f8d2c0069e0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d3cff9640 1 -- 172.21.15.28:0/302207694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3000de00 con 0x7f8d2c0069e0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d40072090 con 0x7f8d2c0069e0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.234+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4004e850 con 0x7f8d2c0069e0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.238+0000 7f8d3cff9640 1 -- 172.21.15.28:0/302207694 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f8d30014400 con 0x7f8d2c0069e0 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.238+0000 7f8d3cff9640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d200664a0 0x7f8d20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:32.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.238+0000 7f8d3cff9640 1 -- 172.21.15.28:0/302207694 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f8d30087250 con 0x7f8d2c0069e0 2023-12-16T21:58:32.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.238+0000 7f8d37fff640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d200664a0 0x7f8d20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:32.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.238+0000 7f8d37fff640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d200664a0 0x7f8d20068960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8d400aea60 tx=0x7f8d280a7070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:32.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.242+0000 7f8d3cff9640 1 -- 172.21.15.28:0/302207694 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f8d30054550 con 0x7f8d2c0069e0 2023-12-16T21:58:32.401 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:32.398+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8d400722b0 con 0x7f8d2c0069e0 2023-12-16T21:58:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.070+0000 7f8d3cff9640 1 -- 172.21.15.28:0/302207694 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v27) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8d300180d0 con 0x7f8d2c0069e0 2023-12-16T21:58:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2023-12-16T21:58:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d200664a0 msgr2=0x7f8d20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d200664a0 0x7f8d20068960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8d400aea60 tx=0x7f8d280a7070 comp rx=0 tx=0).stop 2023-12-16T21:58:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 msgr2=0x7f8d401528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d401528e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000f8c0 tx=0x7f8d3000fd80 comp rx=0 tx=0).stop 2023-12-16T21:58:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 shutdown_connections 2023-12-16T21:58:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8d200664a0 0x7f8d20068960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f8d400b20c0 0x7f8d40070930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d400725b0 0x7f8d400b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 --2- 172.21.15.28:0/302207694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c0069e0 0x7f8d401528e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 >> 172.21.15.28:0/302207694 conn(0x7f8d400b0c50 msgr2=0x7f8d401594c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 shutdown_connections 2023-12-16T21:58:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.074+0000 7f8d45cc8640 1 -- 172.21.15.28:0/302207694 wait complete. 2023-12-16T21:58:33.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-16T21:58:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.230+0000 7f4ba70cc640 1 Processor -- start 2023-12-16T21:58:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.230+0000 7f4ba70cc640 1 -- start start 2023-12-16T21:58:33.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.230+0000 7f4ba70cc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ba006d6f0 0x7f4ba00cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.230+0000 7f4ba70cc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4ba00ce470 0x7f4ba0164e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.230+0000 7f4ba70cc640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba015b6e0 con 0x7f4ba00cce60 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.230+0000 7f4ba70cc640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba00cb550 con 0x7f4ba006d6f0 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.230+0000 7f4ba70cc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba00cb830 con 0x7f4ba00ce470 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.230+0000 7f4b9ffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ba006d6f0 0x7f4ba00cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba4e68640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4ba00cce60 0x7f4ba00cd240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52070/0 (socket says 172.21.15.28:52070) 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba4e68640 1 -- 172.21.15.28:0/4155844815 learned_addr learned my addr 172.21.15.28:0/4155844815 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9ffff640 1 -- 172.21.15.28:0/4155844815 >> v2:172.21.15.122:3300/0 conn(0x7f4ba00ce470 msgr2=0x7f4ba0164e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba5669640 1 --2- 172.21.15.28:0/4155844815 >> v2:172.21.15.122:3300/0 conn(0x7f4ba00ce470 0x7f4ba0164e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9ffff640 1 --2- 172.21.15.28:0/4155844815 >> v2:172.21.15.122:3300/0 conn(0x7f4ba00ce470 0x7f4ba0164e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9f7fe640 1 -- 172.21.15.28:0/4155844815 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069827090 0 0) 0x7f4ba015b6e0 con 0x7f4ba00cce60 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9ffff640 1 -- 172.21.15.28:0/4155844815 >> v1:172.21.15.28:6789/0 conn(0x7f4ba00cce60 legacy=0x7f4ba00cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9ffff640 1 -- 172.21.15.28:0/4155844815 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00cbb10 con 0x7f4ba006d6f0 2023-12-16T21:58:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba5669640 1 --2- 172.21.15.28:0/4155844815 >> v2:172.21.15.122:3300/0 conn(0x7f4ba00ce470 0x7f4ba0164e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9ffff640 1 --2- 172.21.15.28:0/4155844815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba006d6f0 0x7f4ba00cdf30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0070510 tx=0x7f4b9401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2cb9bb73fa05cb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9f7fe640 1 -- 172.21.15.28:0/4155844815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94004230 con 0x7f4ba006d6f0 2023-12-16T21:58:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9f7fe640 1 -- 172.21.15.28:0/4155844815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b940a7050 con 0x7f4ba006d6f0 2023-12-16T21:58:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4b9f7fe640 1 -- 172.21.15.28:0/4155844815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94003620 con 0x7f4ba006d6f0 2023-12-16T21:58:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba70cc640 1 -- 172.21.15.28:0/4155844815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba006d6f0 msgr2=0x7f4ba00cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/4155844815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba006d6f0 0x7f4ba00cdf30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0070510 tx=0x7f4b9401c460 comp rx=0 tx=0).stop 2023-12-16T21:58:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba70cc640 1 -- 172.21.15.28:0/4155844815 shutdown_connections 2023-12-16T21:58:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/4155844815 >> v2:172.21.15.122:3300/0 conn(0x7f4ba00ce470 0x7f4ba0164e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/4155844815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba006d6f0 0x7f4ba00cdf30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba70cc640 1 -- 172.21.15.28:0/4155844815 >> 172.21.15.28:0/4155844815 conn(0x7f4ba00b0c50 msgr2=0x7f4ba00b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba70cc640 1 -- 172.21.15.28:0/4155844815 shutdown_connections 2023-12-16T21:58:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.234+0000 7f4ba70cc640 1 -- 172.21.15.28:0/4155844815 wait complete. 2023-12-16T21:58:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 Processor -- start 2023-12-16T21:58:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 -- start start 2023-12-16T21:58:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba006d6f0 0x7f4ba00b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba00cce60 0x7f4ba0159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4ba00ce470 0x7f4ba015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00b4120 con 0x7f4ba00cce60 2023-12-16T21:58:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba015a0a0 con 0x7f4ba006d6f0 2023-12-16T21:58:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba015c270 con 0x7f4ba00ce470 2023-12-16T21:58:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba00cce60 0x7f4ba0159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba00cce60 0x7f4ba0159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48812/0 (socket says 172.21.15.28:48812) 2023-12-16T21:58:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 -- 172.21.15.28:0/2879664130 learned_addr learned my addr 172.21.15.28:0/2879664130 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba5669640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4ba00ce470 0x7f4ba015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 -- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4ba00ce470 msgr2=0x7f4ba015b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4ba00ce470 0x7f4ba015b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba4e68640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba006d6f0 0x7f4ba00b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 -- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba006d6f0 msgr2=0x7f4ba00b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba006d6f0 0x7f4ba00b3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 -- 172.21.15.28:0/2879664130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b94009420 con 0x7f4ba00cce60 2023-12-16T21:58:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba5669640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4ba00ce470 0x7f4ba015b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9ffff640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba00cce60 0x7f4ba0159b60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4b940ae040 tx=0x7f4b94019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9d7fa640 1 -- 172.21.15.28:0/2879664130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94003c70 con 0x7f4ba00cce60 2023-12-16T21:58:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9d7fa640 1 -- 172.21.15.28:0/2879664130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b94003dd0 con 0x7f4ba00cce60 2023-12-16T21:58:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba00b3100 con 0x7f4ba00cce60 2023-12-16T21:58:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9d7fa640 1 -- 172.21.15.28:0/2879664130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b940ab660 con 0x7f4ba00cce60 2023-12-16T21:58:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba00b36c0 con 0x7f4ba00cce60 2023-12-16T21:58:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba004e850 con 0x7f4ba00cce60 2023-12-16T21:58:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.238+0000 7f4b9d7fa640 1 -- 172.21.15.28:0/2879664130 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4b940b2040 con 0x7f4ba00cce60 2023-12-16T21:58:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.242+0000 7f4b9d7fa640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b8c066470 0x7f4b8c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.242+0000 7f4b9d7fa640 1 -- 172.21.15.28:0/2879664130 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f4b9411de10 con 0x7f4ba00cce60 2023-12-16T21:58:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.242+0000 7f4ba4e68640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b8c066470 0x7f4b8c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:33.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.242+0000 7f4ba4e68640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b8c066470 0x7f4b8c068930 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4b90000f70 tx=0x7f4b90009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:33.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.246+0000 7f4b9d7fa640 1 -- 172.21.15.28:0/2879664130 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4b94121030 con 0x7f4ba00cce60 2023-12-16T21:58:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:33.402+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4ba00ca050 con 0x7f4ba00cce60 2023-12-16T21:58:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.082+0000 7f4b9d7fa640 1 -- 172.21.15.28:0/2879664130 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v28) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4b940b00d0 con 0x7f4ba00cce60 2023-12-16T21:58:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2023-12-16T21:58:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b8c066470 msgr2=0x7f4b8c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b8c066470 0x7f4b8c068930 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4b90000f70 tx=0x7f4b90009040 comp rx=0 tx=0).stop 2023-12-16T21:58:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba00cce60 msgr2=0x7f4ba0159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba00cce60 0x7f4ba0159b60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4b940ae040 tx=0x7f4b94019200 comp rx=0 tx=0).stop 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 shutdown_connections 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b8c066470 0x7f4b8c068930 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4ba00ce470 0x7f4ba015b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ba00cce60 0x7f4ba0159b60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 --2- 172.21.15.28:0/2879664130 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ba006d6f0 0x7f4ba00b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 >> 172.21.15.28:0/2879664130 conn(0x7f4ba00b0c50 msgr2=0x7f4ba00cd240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 shutdown_connections 2023-12-16T21:58:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.086+0000 7f4ba70cc640 1 -- 172.21.15.28:0/2879664130 wait complete. 2023-12-16T21:58:34.120 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-16T21:58:34.121 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-16T21:58:34.121 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-16T21:58:34.121 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:58:34.121 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-16T21:58:34.121 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-16T21:58:34.121 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-16T21:58:34.121 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:58:34.122 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-16T21:58:34.122 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-16T21:58:34.122 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-16T21:58:34.122 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:58:34.122 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-16T21:58:34.122 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-16T21:58:34.122 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-16T21:58:34.122 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:58:34.133 INFO:tasks.ceph.mds.c:Stopped 2023-12-16T21:58:34.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-16T21:58:34.135 INFO:tasks.ceph.mds.a:Stopped 2023-12-16T21:58:34.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-16T21:58:34.140 INFO:tasks.ceph.mds.b:Stopped 2023-12-16T21:58:34.140 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-16T21:58:34.141 INFO:tasks.ceph.mds.d:Stopped 2023-12-16T21:58:34.141 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-16T21:58:34.184 INFO:tasks.ceph.mds.c:Started 2023-12-16T21:58:34.185 DEBUG:teuthology.parallel:result is None 2023-12-16T21:58:34.186 INFO:tasks.ceph.mds.b:Started 2023-12-16T21:58:34.186 INFO:tasks.ceph.mds.a:Started 2023-12-16T21:58:34.186 INFO:tasks.ceph.mds.d:Started 2023-12-16T21:58:34.187 DEBUG:teuthology.parallel:result is None 2023-12-16T21:58:34.187 DEBUG:teuthology.parallel:result is None 2023-12-16T21:58:34.187 DEBUG:teuthology.parallel:result is None 2023-12-16T21:58:34.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-16T21:58:34.257 INFO:tasks.ceph.mds.d.smithi122.stdout:starting mds.d at 2023-12-16T21:58:34.269 INFO:tasks.ceph.mds.b.smithi122.stdout:starting mds.b at 2023-12-16T21:58:34.287 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2023-12-16T21:58:34.303 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2023-12-16T21:58:34.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.338+0000 7f87f8f19640 1 Processor -- start 2023-12-16T21:58:34.343 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.338+0000 7f87f8f19640 1 -- start start 2023-12-16T21:58:34.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f8f19640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f415d160 0x7f87f40aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:34.344 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f8f19640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f87f406b540 0x7f87f406b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f8f19640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f406bed0 con 0x7f87f406cd80 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f8f19640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f406e010 con 0x7f87f406b540 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f8f19640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40aea70 con 0x7f87f415d160 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f1d9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f87f406cd80 0x7f87f40ad5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52072/0 (socket says 172.21.15.28:52072) 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f1d9b640 1 -- 172.21.15.28:0/1543501355 learned_addr learned my addr 172.21.15.28:0/1543501355 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f2d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f87f406b540 0x7f87f406b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f259c640 1 --2- 172.21.15.28:0/1543501355 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f415d160 0x7f87f40aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f159a640 1 -- 172.21.15.28:0/1543501355 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994230999 0 0) 0x7f87dc001120 con 0x7f87f406cd80 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f159a640 1 -- 172.21.15.28:0/1543501355 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e0003530 con 0x7f87f406cd80 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f2d9d640 1 -- 172.21.15.28:0/1543501355 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f415d160 msgr2=0x7f87f40aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f2d9d640 1 --2- 172.21.15.28:0/1543501355 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f415d160 0x7f87f40aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f2d9d640 1 -- 172.21.15.28:0/1543501355 >> v1:172.21.15.28:6789/0 conn(0x7f87f406cd80 legacy=0x7f87f40ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:34.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f2d9d640 1 -- 172.21.15.28:0/1543501355 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40addf0 con 0x7f87f406b540 2023-12-16T21:58:34.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f2d9d640 1 --2- 172.21.15.28:0/1543501355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f406b540 0x7f87f406b990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009d90 tx=0x7f87ec01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c4d4eb97dc75cb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:34.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f159a640 1 -- 172.21.15.28:0/1543501355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0a5870 con 0x7f87f406b540 2023-12-16T21:58:34.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f159a640 1 -- 172.21.15.28:0/1543501355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec0a59d0 con 0x7f87f406b540 2023-12-16T21:58:34.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.342+0000 7f87f159a640 1 -- 172.21.15.28:0/1543501355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0a5d60 con 0x7f87f406b540 2023-12-16T21:58:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- 172.21.15.28:0/1543501355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f406b540 msgr2=0x7f87f406b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 --2- 172.21.15.28:0/1543501355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f406b540 0x7f87f406b990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009d90 tx=0x7f87ec01c8c0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- 172.21.15.28:0/1543501355 shutdown_connections 2023-12-16T21:58:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 --2- 172.21.15.28:0/1543501355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f406b540 0x7f87f406b990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 --2- 172.21.15.28:0/1543501355 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f415d160 0x7f87f40aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- 172.21.15.28:0/1543501355 >> 172.21.15.28:0/1543501355 conn(0x7f87f40b1af0 msgr2=0x7f87f4070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- 172.21.15.28:0/1543501355 shutdown_connections 2023-12-16T21:58:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- 172.21.15.28:0/1543501355 wait complete. 2023-12-16T21:58:34.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 Processor -- start 2023-12-16T21:58:34.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- start start 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f406b540 0x7f87f415b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87f406cd80 0x7f87f415f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f415d160 0x7f87f415f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f414cf80 con 0x7f87f415d160 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41573a0 con 0x7f87f406cd80 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41575f0 con 0x7f87f406b540 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f415d160 0x7f87f415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f415d160 0x7f87f415f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48880/0 (socket says 172.21.15.28:48880) 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 -- 172.21.15.28:0/2308052827 learned_addr learned my addr 172.21.15.28:0/2308052827 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f259c640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f406b540 0x7f87f415b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f1d9b640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87f406cd80 0x7f87f415f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 -- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f406b540 msgr2=0x7f87f415b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f406b540 0x7f87f415b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 -- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87f406cd80 msgr2=0x7f87f415f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87f406cd80 0x7f87f415f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 -- 172.21.15.28:0/2308052827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec009a70 con 0x7f87f415d160 2023-12-16T21:58:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f2d9d640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f415d160 0x7f87f415f550 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0b0040 tx=0x7f87ec019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87eaffd640 1 -- 172.21.15.28:0/2308052827 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0b9650 con 0x7f87f415d160 2023-12-16T21:58:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87eaffd640 1 -- 172.21.15.28:0/2308052827 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec0b97b0 con 0x7f87f415d160 2023-12-16T21:58:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87eaffd640 1 -- 172.21.15.28:0/2308052827 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0b9b60 con 0x7f87f415d160 2023-12-16T21:58:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f41578d0 con 0x7f87f415d160 2023-12-16T21:58:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.346+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f4157e90 con 0x7f87f415d160 2023-12-16T21:58:34.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.350+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f414d0f0 con 0x7f87f415d160 2023-12-16T21:58:34.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.350+0000 7f87eaffd640 1 -- 172.21.15.28:0/2308052827 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f87ec0b9cc0 con 0x7f87f415d160 2023-12-16T21:58:34.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.350+0000 7f87eaffd640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f87d4066470 0x7f87d4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:34.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.350+0000 7f87eaffd640 1 -- 172.21.15.28:0/2308052827 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f87ec11ec10 con 0x7f87f415d160 2023-12-16T21:58:34.353 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.350+0000 7f87f259c640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f87d4066470 0x7f87d4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:34.354 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.350+0000 7f87f259c640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f87d4066470 0x7f87d4068930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009ac0 tx=0x7f87e4009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:34.357 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.354+0000 7f87eaffd640 1 -- 172.21.15.28:0/2308052827 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f87ec0ec4d0 con 0x7f87f415d160 2023-12-16T21:58:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:34.522+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f87f4069440 con 0x7f87f415d160 2023-12-16T21:58:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.110+0000 7f87eaffd640 1 -- 172.21.15.28:0/2308052827 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7f87ec0efaf0 con 0x7f87f415d160 2023-12-16T21:58:35.114 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2023-12-16T21:58:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f87d4066470 msgr2=0x7f87d4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f87d4066470 0x7f87d4068930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009ac0 tx=0x7f87e4009190 comp rx=0 tx=0).stop 2023-12-16T21:58:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f415d160 msgr2=0x7f87f415f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f415d160 0x7f87f415f550 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0b0040 tx=0x7f87ec019660 comp rx=0 tx=0).stop 2023-12-16T21:58:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 shutdown_connections 2023-12-16T21:58:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f87d4066470 0x7f87d4068930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f415d160 0x7f87f415f550 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87f406cd80 0x7f87f415f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 --2- 172.21.15.28:0/2308052827 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f87f406b540 0x7f87f415b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 >> 172.21.15.28:0/2308052827 conn(0x7f87f40b1af0 msgr2=0x7f87f4160aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 shutdown_connections 2023-12-16T21:58:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.114+0000 7f87f8f19640 1 -- 172.21.15.28:0/2308052827 wait complete. 2023-12-16T21:58:35.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-16T21:58:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28c325640 1 Processor -- start 2023-12-16T21:58:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28c325640 1 -- start start 2023-12-16T21:58:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28c325640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28413cf30 0x7fd28406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28c325640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd28415ad40 0x7fd2840cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28c325640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28415b3a0 con 0x7fd28406c800 2023-12-16T21:58:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28c325640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28413d310 con 0x7fd28415ad40 2023-12-16T21:58:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28c325640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28415b1b0 con 0x7fd28413cf30 2023-12-16T21:58:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28a8c2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd28415ad40 0x7fd2840cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28a8c2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd28415ad40 0x7fd2840cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48886/0 (socket says 172.21.15.28:48886) 2023-12-16T21:58:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28a8c2640 1 -- 172.21.15.28:0/3555486 learned_addr learned my addr 172.21.15.28:0/3555486 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28a0c1640 1 --2- 172.21.15.28:0/3555486 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28413cf30 0x7fd28406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28a8c2640 1 -- 172.21.15.28:0/3555486 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28413cf30 msgr2=0x7fd28406c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28a8c2640 1 --2- 172.21.15.28:0/3555486 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28413cf30 0x7fd28406c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28a8c2640 1 -- 172.21.15.28:0/3555486 >> v1:172.21.15.122:6789/0 conn(0x7fd28406c800 legacy=0x7fd28415a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.258+0000 7fd28a8c2640 1 -- 172.21.15.28:0/3555486 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28406cc40 con 0x7fd28415ad40 2023-12-16T21:58:35.263 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 --2- 172.21.15.28:0/3555486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd2840cd880 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009e50 tx=0x7fd27c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50daa59cc1f48feb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:35.263 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd2890bf640 1 -- 172.21.15.28:0/3555486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c0a56b0 con 0x7fd28415ad40 2023-12-16T21:58:35.263 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd2890bf640 1 -- 172.21.15.28:0/3555486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c0a5810 con 0x7fd28415ad40 2023-12-16T21:58:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd2890bf640 1 -- 172.21.15.28:0/3555486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c0a5ba0 con 0x7fd28415ad40 2023-12-16T21:58:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- 172.21.15.28:0/3555486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 msgr2=0x7fd2840cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 --2- 172.21.15.28:0/3555486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd2840cd880 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009e50 tx=0x7fd27c01c940 comp rx=0 tx=0).stop 2023-12-16T21:58:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- 172.21.15.28:0/3555486 shutdown_connections 2023-12-16T21:58:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 --2- 172.21.15.28:0/3555486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd2840cd880 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 --2- 172.21.15.28:0/3555486 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28413cf30 0x7fd28406c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- 172.21.15.28:0/3555486 >> 172.21.15.28:0/3555486 conn(0x7fd28406a530 msgr2=0x7fd28406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- 172.21.15.28:0/3555486 shutdown_connections 2023-12-16T21:58:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- 172.21.15.28:0/3555486 wait complete. 2023-12-16T21:58:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 Processor -- start 2023-12-16T21:58:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- start start 2023-12-16T21:58:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28406c800 0x7fd28413e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd28413cf30 0x7fd2840b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd284149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841489d0 con 0x7fd28415ad40 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28414f550 con 0x7fd28413cf30 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28414dbb0 con 0x7fd28406c800 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd284149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd284149170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48894/0 (socket says 172.21.15.28:48894) 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 -- 172.21.15.28:0/835338087 learned_addr learned my addr 172.21.15.28:0/835338087 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd2898c0640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd28413cf30 0x7fd2840b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a0c1640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28406c800 0x7fd28413e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 -- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28406c800 msgr2=0x7fd28413e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28406c800 0x7fd28413e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 -- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd28413cf30 msgr2=0x7fd2840b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd28413cf30 0x7fd2840b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 -- 172.21.15.28:0/835338087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c009b30 con 0x7fd28415ad40 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a0c1640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28406c800 0x7fd28413e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd2898c0640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd28413cf30 0x7fd2840b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28a8c2640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd284149170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd27c01aff0 tx=0x7fd27c0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd27affd640 1 -- 172.21.15.28:0/835338087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c0b8670 con 0x7fd28415ad40 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd27affd640 1 -- 172.21.15.28:0/835338087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c0b87d0 con 0x7fd28415ad40 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28414de90 con 0x7fd28415ad40 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.262+0000 7fd27affd640 1 -- 172.21.15.28:0/835338087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c0b8b40 con 0x7fd28415ad40 2023-12-16T21:58:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.266+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28414f200 con 0x7fd28415ad40 2023-12-16T21:58:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.266+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28414e0d0 con 0x7fd28415ad40 2023-12-16T21:58:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.266+0000 7fd27affd640 1 -- 172.21.15.28:0/835338087 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fd27c0b8ca0 con 0x7fd28415ad40 2023-12-16T21:58:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.266+0000 7fd27affd640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd268066470 0x7fd268068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.266+0000 7fd27affd640 1 -- 172.21.15.28:0/835338087 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd27c11f0b0 con 0x7fd28415ad40 2023-12-16T21:58:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.266+0000 7fd28a0c1640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd268066470 0x7fd268068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.266+0000 7fd28a0c1640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd268066470 0x7fd268068930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd280006fd0 tx=0x7fd280008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:35.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.270+0000 7fd27affd640 1 -- 172.21.15.28:0/835338087 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fd27c0ec880 con 0x7fd28415ad40 2023-12-16T21:58:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.474+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd284069440 con 0x7fd28415ad40 2023-12-16T21:58:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.474+0000 7fd27affd640 1 -- 172.21.15.28:0/835338087 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fd27c0efea0 con 0x7fd28415ad40 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAdHX5lNhVfOBAA/8ufBguEtvKEzSvyiAM76A==", 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.479 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAeHX5lUYlUCxAAeDMP2IT7cmcsa4/6kRuw2A==", 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAeHX5lKTRaBRAAeeuQC4/6XmX+rr4kF8hW6g==", 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAeHX5lOGooFBAAY3NgNlKKyMU91K87IMa6ig==", 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.481 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAfHX5lk1STKhAACY4gGmaxvS+gtbtS/YaLvw==", 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAiHX5lvCNjMRAAKFid+EnG5Kn3HL2c08AMcw==", 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAlHX5lPycMOhAATI0Bl+hIinidrRZ+gavRVQ==", 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2023-12-16T21:58:35.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQApHX5lx+iGBxAAJxGIuBGMpJlsm29neKGMFQ==", 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2023-12-16T21:58:35.484 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAvHX5l4fTYARAA5qFsU8D21FyFc4kIiRwJ7A==", 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAyHX5l0CV7ChAA6LWW7LQC4ZVGrWRoVHCpzQ==", 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA1HX5lnxBaFBAAl6/cEncJHyna8B1kWm0ZFA==", 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA4HX5la9GcHRAAA720RxZp4IkNmVKnie1Ehg==", 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2023-12-16T21:58:35.486 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAeHX5l+MjmGRAAJRtEVtccO6nez+DbuubEww==", 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAeHX5lBSu0IRAAzzxW4dkqOsC0xhF4RfzG/g==", 2023-12-16T21:58:35.487 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAeHX5l7ZT+HhAAqOvNU4UQ8KmT7hw2/NfVww==", 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2023-12-16T21:58:35.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAeHX5ltiOWJhAAtNM2X01MKH4/7p7QdHO/zg==", 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2023-12-16T21:58:35.489 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAdHX5lX91pIRAARh7B+7emWJR5gGjk5UAYTw==", 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mds", 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA+HX5ljJEnKxAAa3Q5fcztZPbThzkFWeUnGw==", 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mds" 2023-12-16T21:58:35.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mgr", 2023-12-16T21:58:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA+HX5lZKUnKxAAUSG/fwI6zB2lgd6G3nsqZA==", 2023-12-16T21:58:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.492 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mgr" 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-osd", 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA+HX5lvLYnKxAA3ne16/eOkgHXIolttHLIFw==", 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-osd" 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd", 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA+HX5l78cnKxAArLFPFVc9xol3mc9grySGFA==", 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd" 2023-12-16T21:58:35.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA+HX5llNknKxAArVQJ5Hfs9oCf/fRKVuIXHA==", 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rgw", 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA+HX5lmesnKxAAifCmOuzAED4fb4lTSWhuAA==", 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rgw" 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAdHX5lCe/6NBAAgSGT10P74RECk7ODutoPtw==", 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQAdHX5l4hVQMhAA4+5NWEbViUXc1iOKVBLf/A==", 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2023-12-16T21:58:35.495 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2023-12-16T21:58:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2023-12-16T21:58:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2023-12-16T21:58:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:35.496 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2023-12-16T21:58:35.496 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-16T21:58:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd268066470 msgr2=0x7fd268068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd268066470 0x7fd268068930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd280006fd0 tx=0x7fd280008040 comp rx=0 tx=0).stop 2023-12-16T21:58:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 msgr2=0x7fd284149170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd284149170 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd27c01aff0 tx=0x7fd27c0196e0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 shutdown_connections 2023-12-16T21:58:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd268066470 0x7fd268068930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28415ad40 0x7fd284149170 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd28413cf30 0x7fd2840b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 --2- 172.21.15.28:0/835338087 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd28406c800 0x7fd28413e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 >> 172.21.15.28:0/835338087 conn(0x7fd28406a530 msgr2=0x7fd284071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 shutdown_connections 2023-12-16T21:58:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.478+0000 7fd28c325640 1 -- 172.21.15.28:0/835338087 wait complete. 2023-12-16T21:58:35.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-12-16T21:58:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.618+0000 7f4b893ff640 1 Processor -- start 2023-12-16T21:58:35.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.618+0000 7f4b893ff640 1 -- start start 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.618+0000 7f4b893ff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8406c9d0 0x7f4b84070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.618+0000 7f4b893ff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b84072180 0x7f4b840709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.618+0000 7f4b893ff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b840710e0 con 0x7f4b84144840 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.618+0000 7f4b893ff640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8415f010 con 0x7f4b84072180 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.618+0000 7f4b893ff640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84070ee0 con 0x7f4b8406c9d0 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b837fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b84072180 0x7f4b840709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b837fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b84072180 0x7f4b840709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48908/0 (socket says 172.21.15.28:48908) 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b837fe640 1 -- 172.21.15.28:0/3540369798 learned_addr learned my addr 172.21.15.28:0/3540369798 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b827fc640 1 --2- 172.21.15.28:0/3540369798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8406c9d0 0x7f4b84070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b837fe640 1 -- 172.21.15.28:0/3540369798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8406c9d0 msgr2=0x7f4b84070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b837fe640 1 --2- 172.21.15.28:0/3540369798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8406c9d0 0x7f4b84070460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b837fe640 1 -- 172.21.15.28:0/3540369798 >> v1:172.21.15.122:6789/0 conn(0x7f4b84144840 legacy=0x7f4b8406c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b837fe640 1 -- 172.21.15.28:0/3540369798 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b840713c0 con 0x7f4b84072180 2023-12-16T21:58:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b827fc640 1 --2- 172.21.15.28:0/3540369798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8406c9d0 0x7f4b84070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b837fe640 1 --2- 172.21.15.28:0/3540369798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b84072180 0x7f4b840709a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009870 tx=0x7f4b7c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdafc1b0e4eef3a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b81ffb640 1 -- 172.21.15.28:0/3540369798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0aa5f0 con 0x7f4b84072180 2023-12-16T21:58:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b81ffb640 1 -- 172.21.15.28:0/3540369798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c0aa750 con 0x7f4b84072180 2023-12-16T21:58:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b81ffb640 1 -- 172.21.15.28:0/3540369798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0aaae0 con 0x7f4b84072180 2023-12-16T21:58:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b893ff640 1 -- 172.21.15.28:0/3540369798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b84072180 msgr2=0x7f4b840709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b893ff640 1 --2- 172.21.15.28:0/3540369798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b84072180 0x7f4b840709a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009870 tx=0x7f4b7c01c4d0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b893ff640 1 -- 172.21.15.28:0/3540369798 shutdown_connections 2023-12-16T21:58:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b893ff640 1 --2- 172.21.15.28:0/3540369798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b84072180 0x7f4b840709a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b893ff640 1 --2- 172.21.15.28:0/3540369798 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8406c9d0 0x7f4b84070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b893ff640 1 -- 172.21.15.28:0/3540369798 >> 172.21.15.28:0/3540369798 conn(0x7f4b8406a530 msgr2=0x7f4b8406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b893ff640 1 -- 172.21.15.28:0/3540369798 shutdown_connections 2023-12-16T21:58:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.622+0000 7f4b893ff640 1 -- 172.21.15.28:0/3540369798 wait complete. 2023-12-16T21:58:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 Processor -- start 2023-12-16T21:58:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 -- start start 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8406c9d0 0x7f4b8413c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b84072180 0x7f4b84139660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b84144840 0x7f4b84139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b84139270 con 0x7f4b8406c9d0 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8413cc30 con 0x7f4b84072180 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8413d010 con 0x7f4b84144840 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8406c9d0 0x7f4b8413c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8406c9d0 0x7f4b8413c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48910/0 (socket says 172.21.15.28:48910) 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 -- 172.21.15.28:0/2998196550 learned_addr learned my addr 172.21.15.28:0/2998196550 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b837fe640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b84144840 0x7f4b84139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b827fc640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b84072180 0x7f4b84139660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 -- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b84144840 msgr2=0x7f4b84139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b84144840 0x7f4b84139ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 -- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b84072180 msgr2=0x7f4b84139660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b84072180 0x7f4b84139660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 -- 172.21.15.28:0/2998196550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0094d0 con 0x7f4b8406c9d0 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b827fc640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b84072180 0x7f4b84139660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b82ffd640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8406c9d0 0x7f4b8413c650 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400ba50 tx=0x7f4b7400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b6b7fe640 1 -- 172.21.15.28:0/2998196550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7400cbe0 con 0x7f4b8406c9d0 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b6b7fe640 1 -- 172.21.15.28:0/2998196550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7400cd80 con 0x7f4b8406c9d0 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b6b7fe640 1 -- 172.21.15.28:0/2998196550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b74014680 con 0x7f4b8406c9d0 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8413d2f0 con 0x7f4b8406c9d0 2023-12-16T21:58:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8415aad0 con 0x7f4b8406c9d0 2023-12-16T21:58:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.626+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8413a0e0 con 0x7f4b8406c9d0 2023-12-16T21:58:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.630+0000 7f4b6b7fe640 1 -- 172.21.15.28:0/2998196550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4b740147e0 con 0x7f4b8406c9d0 2023-12-16T21:58:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.630+0000 7f4b6b7fe640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.630+0000 7f4b6b7fe640 1 -- 172.21.15.28:0/2998196550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4b74083a60 con 0x7f4b8406c9d0 2023-12-16T21:58:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.630+0000 7f4b827fc640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.630+0000 7f4b827fc640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4b841f9400 tx=0x7f4b78008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.634+0000 7f4b6b7fe640 1 -- 172.21.15.28:0/2998196550 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4b74051240 con 0x7f4b8406c9d0 2023-12-16T21:58:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.846+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f4b84069440 con 0x7f4b8406c9d0 2023-12-16T21:58:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.850+0000 7f4b6b7fe640 1 -- 172.21.15.28:0/2998196550 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4b74054860 con 0x7f4b8406c9d0 2023-12-16T21:58:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 msgr2=0x7f4b70068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4b841f9400 tx=0x7f4b78008040 comp rx=0 tx=0).stop 2023-12-16T21:58:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8406c9d0 msgr2=0x7f4b8413c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8406c9d0 0x7f4b8413c650 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400ba50 tx=0x7f4b7400bf10 comp rx=0 tx=0).stop 2023-12-16T21:58:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 shutdown_connections 2023-12-16T21:58:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b84144840 0x7f4b84139ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b84072180 0x7f4b84139660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 --2- 172.21.15.28:0/2998196550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8406c9d0 0x7f4b8413c650 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 >> 172.21.15.28:0/2998196550 conn(0x7f4b8406a530 msgr2=0x7f4b84143880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 shutdown_connections 2023-12-16T21:58:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.854+0000 7f4b893ff640 1 -- 172.21.15.28:0/2998196550 wait complete. 2023-12-16T21:58:35.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-12-16T21:58:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecaef09640 1 Processor -- start 2023-12-16T21:58:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecaef09640 1 -- start start 2023-12-16T21:58:36.001 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecaef09640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feca80b35b0 0x7feca80d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.001 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecaef09640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7feca80b3c70 0x7feca815f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.001 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecaef09640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca80d3880 con 0x7feca80cf780 2023-12-16T21:58:36.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecaef09640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca80d4090 con 0x7feca80b35b0 2023-12-16T21:58:36.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecaef09640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca80cdc80 con 0x7feca80b3c70 2023-12-16T21:58:36.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecacca5640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feca80b35b0 0x7feca80d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecad4a6640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feca80cf780 0x7feca80ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52082/0 (socket says 172.21.15.28:52082) 2023-12-16T21:58:36.002 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecad4a6640 1 -- 172.21.15.28:0/2328965831 learned_addr learned my addr 172.21.15.28:0/2328965831 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7feca7fff640 1 --2- 172.21.15.28:0/2328965831 >> v2:172.21.15.122:3300/0 conn(0x7feca80b3c70 0x7feca815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecacca5640 1 -- 172.21.15.28:0/2328965831 >> v2:172.21.15.122:3300/0 conn(0x7feca80b3c70 msgr2=0x7feca815f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecacca5640 1 --2- 172.21.15.28:0/2328965831 >> v2:172.21.15.122:3300/0 conn(0x7feca80b3c70 0x7feca815f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7feca77fe640 1 -- 172.21.15.28:0/2328965831 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340147200 0 0) 0x7feca80d3880 con 0x7feca80cf780 2023-12-16T21:58:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecacca5640 1 -- 172.21.15.28:0/2328965831 >> v1:172.21.15.28:6789/0 conn(0x7feca80cf780 legacy=0x7feca80ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7fecacca5640 1 -- 172.21.15.28:0/2328965831 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca80d3ad0 con 0x7feca80b35b0 2023-12-16T21:58:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:35.998+0000 7feca7fff640 1 --2- 172.21.15.28:0/2328965831 >> v2:172.21.15.122:3300/0 conn(0x7feca80b3c70 0x7feca815f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecacca5640 1 --2- 172.21.15.28:0/2328965831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b35b0 0x7feca80d0f50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feca808bde0 tx=0x7fec9801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29c3966728a7d78 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7feca77fe640 1 -- 172.21.15.28:0/2328965831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec980ab660 con 0x7feca80b35b0 2023-12-16T21:58:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7feca77fe640 1 -- 172.21.15.28:0/2328965831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec980ab7c0 con 0x7feca80b35b0 2023-12-16T21:58:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7feca77fe640 1 -- 172.21.15.28:0/2328965831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec980abb50 con 0x7feca80b35b0 2023-12-16T21:58:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecaef09640 1 -- 172.21.15.28:0/2328965831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b35b0 msgr2=0x7feca80d0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecaef09640 1 --2- 172.21.15.28:0/2328965831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b35b0 0x7feca80d0f50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feca808bde0 tx=0x7fec9801c460 comp rx=0 tx=0).stop 2023-12-16T21:58:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecaef09640 1 -- 172.21.15.28:0/2328965831 shutdown_connections 2023-12-16T21:58:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecaef09640 1 --2- 172.21.15.28:0/2328965831 >> v2:172.21.15.122:3300/0 conn(0x7feca80b3c70 0x7feca815f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecaef09640 1 --2- 172.21.15.28:0/2328965831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b35b0 0x7feca80d0f50 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecaef09640 1 -- 172.21.15.28:0/2328965831 >> 172.21.15.28:0/2328965831 conn(0x7feca809c4b0 msgr2=0x7feca80d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecaef09640 1 -- 172.21.15.28:0/2328965831 shutdown_connections 2023-12-16T21:58:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.002+0000 7fecaef09640 1 -- 172.21.15.28:0/2328965831 wait complete. 2023-12-16T21:58:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 Processor -- start 2023-12-16T21:58:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 -- start start 2023-12-16T21:58:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feca80b35b0 0x7feca80b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b3c70 0x7feca80c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feca80cf780 0x7feca80c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca8148ee0 con 0x7feca80b3c70 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca8148c00 con 0x7feca80b35b0 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca80b4f40 con 0x7feca80cf780 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b3c70 0x7feca80c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b3c70 0x7feca80c8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48918/0 (socket says 172.21.15.28:48918) 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 -- 172.21.15.28:0/1771117926 learned_addr learned my addr 172.21.15.28:0/1771117926 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecacca5640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feca80b35b0 0x7feca80b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 -- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feca80cf780 msgr2=0x7feca80c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feca80cf780 0x7feca80c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 -- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feca80b35b0 msgr2=0x7feca80b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feca80b35b0 0x7feca80b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 -- 172.21.15.28:0/1771117926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec98009420 con 0x7feca80b3c70 2023-12-16T21:58:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecacca5640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feca80b35b0 0x7feca80b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca7fff640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b3c70 0x7feca80c8930 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fec9400c7c0 tx=0x7fec9400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca57fa640 1 -- 172.21.15.28:0/1771117926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94007a30 con 0x7feca80b3c70 2023-12-16T21:58:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca57fa640 1 -- 172.21.15.28:0/1771117926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec94007b90 con 0x7feca80b3c70 2023-12-16T21:58:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca80d0120 con 0x7feca80b3c70 2023-12-16T21:58:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca57fa640 1 -- 172.21.15.28:0/1771117926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9400ce70 con 0x7feca80b3c70 2023-12-16T21:58:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca80b5500 con 0x7feca80b3c70 2023-12-16T21:58:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca80b4da0 con 0x7feca80b3c70 2023-12-16T21:58:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca57fa640 1 -- 172.21.15.28:0/1771117926 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fec940193f0 con 0x7feca80b3c70 2023-12-16T21:58:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca57fa640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fec90066470 0x7fec90068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.006+0000 7feca57fa640 1 -- 172.21.15.28:0/1771117926 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fec94084020 con 0x7feca80b3c70 2023-12-16T21:58:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.010+0000 7fecacca5640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fec90066470 0x7fec90068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.010+0000 7fecacca5640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fec90066470 0x7fec90068930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fec980af000 tx=0x7fec98008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.015 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.010+0000 7feca57fa640 1 -- 172.21.15.28:0/1771117926 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7feca80b4da0 con 0x7feca80b3c70 2023-12-16T21:58:36.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.221+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7feca80b4da0 con 0x7feca80b3c70 2023-12-16T21:58:36.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7feca57fa640 1 -- 172.21.15.28:0/1771117926 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7feca80b4da0 con 0x7feca80b3c70 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fec90066470 msgr2=0x7fec90068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fec90066470 0x7fec90068930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fec980af000 tx=0x7fec98008040 comp rx=0 tx=0).stop 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b3c70 msgr2=0x7feca80c8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b3c70 0x7feca80c8930 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fec9400c7c0 tx=0x7fec9400cc80 comp rx=0 tx=0).stop 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 shutdown_connections 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fec90066470 0x7fec90068930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7feca80cf780 0x7feca80c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feca80b3c70 0x7feca80c8930 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 --2- 172.21.15.28:0/1771117926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feca80b35b0 0x7feca80b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 >> 172.21.15.28:0/1771117926 conn(0x7feca809c4b0 msgr2=0x7feca806b310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 shutdown_connections 2023-12-16T21:58:36.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.229+0000 7fecaef09640 1 -- 172.21.15.28:0/1771117926 wait complete. 2023-12-16T21:58:36.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-12-16T21:58:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f19759ee640 1 Processor -- start 2023-12-16T21:58:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f19759ee640 1 -- start start 2023-12-16T21:58:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f19759ee640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f197015ad40 0x7f1970164df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f19759ee640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197015b170 con 0x7f197006c9d0 2023-12-16T21:58:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f19759ee640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f197015b3a0 con 0x7f197013cf30 2023-12-16T21:58:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f19759ee640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197013d310 con 0x7f197015ad40 2023-12-16T21:58:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f196e7fc640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f197006c9d0 0x7f197015a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52084/0 (socket says 172.21.15.28:52084) 2023-12-16T21:58:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f196e7fc640 1 -- 172.21.15.28:0/2134238946 learned_addr learned my addr 172.21.15.28:0/2134238946 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.361+0000 7f196f7fe640 1 --2- 172.21.15.28:0/2134238946 >> v2:172.21.15.122:3300/0 conn(0x7f197015ad40 0x7f1970164df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574924755 0 0) 0x7f197015b170 con 0x7f197006c9d0 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1958003530 con 0x7f197006c9d0 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 475304104 0 0) 0x7f1958003530 con 0x7f197006c9d0 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 >> v2:172.21.15.122:3300/0 conn(0x7f197015ad40 msgr2=0x7f1970164df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 --2- 172.21.15.28:0/2134238946 >> v2:172.21.15.122:3300/0 conn(0x7f197015ad40 0x7f1970164df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 >> v1:172.21.15.122:6789/0 conn(0x7f197013cf30 legacy=0x7f197006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197013f720 con 0x7f197006c9d0 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f1960001680 con 0x7f197006c9d0 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 >> v1:172.21.15.28:6789/0 conn(0x7f197006c9d0 legacy=0x7f197015a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1958003980 0x7f1958003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1958004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1958005190 0x7f195800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195800c100 con 0x7f19580047d0 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195800c270 con 0x7f1958005190 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195800c430 con 0x7f1958003980 2023-12-16T21:58:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196e7fc640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1958004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196effd640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1958003980 0x7f1958003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196f7fe640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1958005190 0x7f195800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196e7fc640 1 -- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1958003980 msgr2=0x7f1958003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196e7fc640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1958003980 0x7f1958003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196e7fc640 1 -- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1958005190 msgr2=0x7f195800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196e7fc640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1958005190 0x7f195800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196e7fc640 1 -- 172.21.15.28:0/2134238946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1958003530 con 0x7f19580047d0 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196f7fe640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1958005190 0x7f195800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196f7fe640 1 -- 172.21.15.28:0/2134238946 reap_dead start 2023-12-16T21:58:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196e7fc640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1958004c50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f196000c240 tx=0x7f196001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4949743045b177a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960005ae0 con 0x7f19580047d0 2023-12-16T21:58:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1960005c60 con 0x7f19580047d0 2023-12-16T21:58:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f196dffb640 1 -- 172.21.15.28:0/2134238946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600a5840 con 0x7f19580047d0 2023-12-16T21:58:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f19759ee640 1 -- 172.21.15.28:0/2134238946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 msgr2=0x7f1958004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f19759ee640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1958004c50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f196000c240 tx=0x7f196001cc00 comp rx=0 tx=0).stop 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f19759ee640 1 -- 172.21.15.28:0/2134238946 shutdown_connections 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f19759ee640 1 --2- 172.21.15.28:0/2134238946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1958004c50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f19759ee640 1 -- 172.21.15.28:0/2134238946 >> 172.21.15.28:0/2134238946 conn(0x7f197006a530 msgr2=0x7f197006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f19759ee640 1 -- 172.21.15.28:0/2134238946 shutdown_connections 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.365+0000 7f19759ee640 1 -- 172.21.15.28:0/2134238946 wait complete. 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 Processor -- start 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 -- start start 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1970168f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1970134900 0x7f19700a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1970134010 0x7f197014f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19700b0580 con 0x7f19580047d0 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19700af010 con 0x7f1970134900 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19700afaa0 con 0x7f1970134010 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1970168f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1970168f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48936/0 (socket says 172.21.15.28:48936) 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 -- 172.21.15.28:0/1092015080 learned_addr learned my addr 172.21.15.28:0/1092015080 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196f7fe640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1970134010 0x7f197014f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196e7fc640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1970134900 0x7f19700a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 -- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1970134010 msgr2=0x7f197014f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1970134010 0x7f197014f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 -- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1970134900 msgr2=0x7f19700a8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1970134900 0x7f19700a8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 -- 172.21.15.28:0/1092015080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700afd80 con 0x7f19580047d0 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196e7fc640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1970134900 0x7f19700a8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f196effd640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1970168f40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f196400e4b0 tx=0x7f196400e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f1957fff640 1 -- 172.21.15.28:0/1092015080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19640052e0 con 0x7f19580047d0 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f1957fff640 1 -- 172.21.15.28:0/1092015080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1964005440 con 0x7f19580047d0 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19700b0060 con 0x7f19580047d0 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f1957fff640 1 -- 172.21.15.28:0/1092015080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964003840 con 0x7f19580047d0 2023-12-16T21:58:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19700aec30 con 0x7f19580047d0 2023-12-16T21:58:36.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.369+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1970069440 con 0x7f19580047d0 2023-12-16T21:58:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.373+0000 7f1957fff640 1 -- 172.21.15.28:0/1092015080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f19640039a0 con 0x7f19580047d0 2023-12-16T21:58:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.373+0000 7f1957fff640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f195c066470 0x7f195c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.373+0000 7f1957fff640 1 -- 172.21.15.28:0/1092015080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1964083800 con 0x7f19580047d0 2023-12-16T21:58:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.373+0000 7f196e7fc640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f195c066470 0x7f195c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.373+0000 7f196e7fc640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f195c066470 0x7f195c068930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f19600ab000 tx=0x7f19600a7070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.377+0000 7f1957fff640 1 -- 172.21.15.28:0/1092015080 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f1964051060 con 0x7f19580047d0 2023-12-16T21:58:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.585+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f197006d1e0 con 0x7f19580047d0 2023-12-16T21:58:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f1957fff640 1 -- 172.21.15.28:0/1092015080 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1964054680 con 0x7f19580047d0 2023-12-16T21:58:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f195c066470 msgr2=0x7f195c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f195c066470 0x7f195c068930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f19600ab000 tx=0x7f19600a7070 comp rx=0 tx=0).stop 2023-12-16T21:58:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 msgr2=0x7f1970168f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1970168f40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f196400e4b0 tx=0x7f196400e970 comp rx=0 tx=0).stop 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 shutdown_connections 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f195c066470 0x7f195c068930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1970134010 0x7f197014f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1970134900 0x7f19700a8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 --2- 172.21.15.28:0/1092015080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19580047d0 0x7f1970168f40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 >> 172.21.15.28:0/1092015080 conn(0x7f197006a530 msgr2=0x7f1970158380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 shutdown_connections 2023-12-16T21:58:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.589+0000 7f19759ee640 1 -- 172.21.15.28:0/1092015080 wait complete. 2023-12-16T21:58:36.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-12-16T21:58:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7f6fd640 1 Processor -- start 2023-12-16T21:58:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7f6fd640 1 -- start start 2023-12-16T21:58:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7f6fd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f2d7815ad40 0x7f2d78164df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7f6fd640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7815b170 con 0x7f2d7813cf30 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7f6fd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d7815b3a0 con 0x7f2d7806c9d0 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7f6fd640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7813d310 con 0x7f2d7815ad40 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7d499640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2d7813cf30 0x7f2d7806c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52094/0 (socket says 172.21.15.28:52094) 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7d499640 1 -- 172.21.15.28:0/2152759251 learned_addr learned my addr 172.21.15.28:0/2152759251 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7dc9a640 1 --2- 172.21.15.28:0/2152759251 >> v2:172.21.15.122:3300/0 conn(0x7f2d7815ad40 0x7f2d78164df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115070510 0 0) 0x7f2d7815b170 con 0x7f2d7813cf30 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d64003530 con 0x7f2d7813cf30 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3824918943 0 0) 0x7f2d64003530 con 0x7f2d7813cf30 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 >> v2:172.21.15.122:3300/0 conn(0x7f2d7815ad40 msgr2=0x7f2d78164df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 --2- 172.21.15.28:0/2152759251 >> v2:172.21.15.122:3300/0 conn(0x7f2d7815ad40 0x7f2d78164df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 >> v1:172.21.15.122:6789/0 conn(0x7f2d7806c9d0 legacy=0x7f2d7815a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7813f720 con 0x7f2d7813cf30 2023-12-16T21:58:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f2d6c001680 con 0x7f2d7813cf30 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 >> v1:172.21.15.28:6789/0 conn(0x7f2d7813cf30 legacy=0x7f2d7806c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d64003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d64004380 0x7f2d64004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d640050f0 0x7f2d6400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6400c000 con 0x7f2d64003980 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6400c170 con 0x7f2d640050f0 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6400c330 con 0x7f2d64004380 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7d499640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d64003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.721+0000 7f2d7cc98640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d64004380 0x7f2d64004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7dc9a640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d640050f0 0x7f2d6400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7d499640 1 -- 172.21.15.28:0/2152759251 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d64004380 msgr2=0x7f2d64004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7d499640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d64004380 0x7f2d64004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7d499640 1 -- 172.21.15.28:0/2152759251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d640050f0 msgr2=0x7f2d6400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7d499640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d640050f0 0x7f2d6400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7d499640 1 -- 172.21.15.28:0/2152759251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d64003530 con 0x7f2d64003980 2023-12-16T21:58:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7dc9a640 1 -- 172.21.15.28:0/2152759251 reap_dead start 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7d499640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d64003d60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00c210 tx=0x7f2d6c01cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78df206b1e55b17 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c005ae0 con 0x7f2d64003980 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c005c60 con 0x7f2d64003980 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d6bfff640 1 -- 172.21.15.28:0/2152759251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0a5af0 con 0x7f2d64003980 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/2152759251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 msgr2=0x7f2d64003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7f6fd640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d64003d60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00c210 tx=0x7f2d6c01cbd0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/2152759251 shutdown_connections 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7f6fd640 1 --2- 172.21.15.28:0/2152759251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d64003d60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/2152759251 >> 172.21.15.28:0/2152759251 conn(0x7f2d7806a530 msgr2=0x7f2d7806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/2152759251 shutdown_connections 2023-12-16T21:58:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.725+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/2152759251 wait complete. 2023-12-16T21:58:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 Processor -- start 2023-12-16T21:58:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 -- start start 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d780b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d781f4f20 0x7f2d781f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d780ac840 0x7f2d780accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d781f58e0 con 0x7f2d64003980 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780ad230 con 0x7f2d781f4f20 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780ad4c0 con 0x7f2d780ac840 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d780b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d780b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48964/0 (socket says 172.21.15.28:48964) 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 -- 172.21.15.28:0/270826842 learned_addr learned my addr 172.21.15.28:0/270826842 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7dc9a640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d780ac840 0x7f2d780accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 -- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d780ac840 msgr2=0x7f2d780accf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7cc98640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d781f4f20 0x7f2d781f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d780ac840 0x7f2d780accf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 -- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d781f4f20 msgr2=0x7f2d781f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d781f4f20 0x7f2d781f53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 -- 172.21.15.28:0/270826842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d780ad7a0 con 0x7f2d64003980 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7dc9a640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d780ac840 0x7f2d780accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7d499640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d780b2fd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c01b2d0 tx=0x7f2d64005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d69ffb640 1 -- 172.21.15.28:0/270826842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0b4070 con 0x7f2d64003980 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d69ffb640 1 -- 172.21.15.28:0/270826842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c0a58b0 con 0x7f2d64003980 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7813d310 con 0x7f2d64003980 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d69ffb640 1 -- 172.21.15.28:0/270826842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0a5c60 con 0x7f2d64003980 2023-12-16T21:58:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7813d8d0 con 0x7f2d64003980 2023-12-16T21:58:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.729+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d78069440 con 0x7f2d64003980 2023-12-16T21:58:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.733+0000 7f2d69ffb640 1 -- 172.21.15.28:0/270826842 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f2d6c007070 con 0x7f2d64003980 2023-12-16T21:58:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.733+0000 7f2d69ffb640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2d5c0664a0 0x7f2d5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.733+0000 7f2d69ffb640 1 -- 172.21.15.28:0/270826842 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2d6c11d8a0 con 0x7f2d64003980 2023-12-16T21:58:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.733+0000 7f2d7cc98640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2d5c0664a0 0x7f2d5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.733+0000 7f2d7cc98640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2d5c0664a0 0x7f2d5c068960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2d74007fd0 tx=0x7f2d74007a60 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:36.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.737+0000 7f2d69ffb640 1 -- 172.21.15.28:0/270826842 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f2d6c0eb150 con 0x7f2d64003980 2023-12-16T21:58:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.945+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f2d7813c410 con 0x7f2d64003980 2023-12-16T21:58:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d69ffb640 1 -- 172.21.15.28:0/270826842 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f2d6c0ee770 con 0x7f2d64003980 2023-12-16T21:58:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2d5c0664a0 msgr2=0x7f2d5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2d5c0664a0 0x7f2d5c068960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2d74007fd0 tx=0x7f2d74007a60 comp rx=0 tx=0).stop 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 msgr2=0x7f2d780b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d780b2fd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c01b2d0 tx=0x7f2d64005800 comp rx=0 tx=0).stop 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 shutdown_connections 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2d5c0664a0 0x7f2d5c068960 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2d780ac840 0x7f2d780accf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d781f4f20 0x7f2d781f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 --2- 172.21.15.28:0/270826842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d64003980 0x7f2d780b2fd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 >> 172.21.15.28:0/270826842 conn(0x7f2d7806a530 msgr2=0x7f2d78138920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 shutdown_connections 2023-12-16T21:58:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:36.949+0000 7f2d7f6fd640 1 -- 172.21.15.28:0/270826842 wait complete. 2023-12-16T21:58:36.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-12-16T21:58:37.096 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe47dee1640 1 Processor -- start 2023-12-16T21:58:37.096 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe47dee1640 1 -- start start 2023-12-16T21:58:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe47dee1640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe47815d4e0 0x7fe4780cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe47dee1640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47806ddb0 con 0x7fe47806cd10 2023-12-16T21:58:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe47dee1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47806d1a0 con 0x7fe4780713c0 2023-12-16T21:58:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe47dee1640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47815d8c0 con 0x7fe47815d4e0 2023-12-16T21:58:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe46ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe47806cd10 0x7fe478070cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52098/0 (socket says 172.21.15.28:52098) 2023-12-16T21:58:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe46ffff640 1 -- 172.21.15.28:0/3889095397 learned_addr learned my addr 172.21.15.28:0/3889095397 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe4777fe640 1 --2- 172.21.15.28:0/3889095397 >> v2:172.21.15.122:3300/0 conn(0x7fe47815d4e0 0x7fe4780cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228676231 0 0) 0x7fe47806ddb0 con 0x7fe47806cd10 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe464003530 con 0x7fe47806cd10 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145328981 0 0) 0x7fe47806d1a0 con 0x7fe4780713c0 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe47806ddb0 con 0x7fe4780713c0 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3908001646 0 0) 0x7fe464003530 con 0x7fe47806cd10 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 >> v2:172.21.15.122:3300/0 conn(0x7fe47815d4e0 msgr2=0x7fe4780cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 --2- 172.21.15.28:0/3889095397 >> v2:172.21.15.122:3300/0 conn(0x7fe47815d4e0 0x7fe4780cd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 >> v1:172.21.15.122:6789/0 conn(0x7fe4780713c0 legacy=0x7fe47806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47806d480 con 0x7fe47806cd10 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fe460001680 con 0x7fe47806cd10 2023-12-16T21:58:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 >> v1:172.21.15.28:6789/0 conn(0x7fe47806cd10 legacy=0x7fe478070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464003b10 0x7fe464003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4640047d0 0x7fe464004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe464005190 0x7fe46400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47815d8c0 con 0x7fe464003b10 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46400c010 con 0x7fe4640047d0 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46400c1d0 con 0x7fe464005190 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe4777fe640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464003b10 0x7fe464003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe46ffff640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4640047d0 0x7fe464004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe477fff640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe464005190 0x7fe46400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe4777fe640 1 -- 172.21.15.28:0/3889095397 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe464005190 msgr2=0x7fe46400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe4777fe640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe464005190 0x7fe46400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe4777fe640 1 -- 172.21.15.28:0/3889095397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4640047d0 msgr2=0x7fe464004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe4777fe640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4640047d0 0x7fe464004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe4777fe640 1 -- 172.21.15.28:0/3889095397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe464003530 con 0x7fe464003b10 2023-12-16T21:58:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.093+0000 7fe477fff640 1 -- 172.21.15.28:0/3889095397 reap_dead start 2023-12-16T21:58:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.097+0000 7fe4777fe640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464003b10 0x7fe464003f10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe468009ae0 tx=0x7fe46801ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d94faf6c97e56fd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.097+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4680a6500 con 0x7fe464003b10 2023-12-16T21:58:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.097+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4680a6660 con 0x7fe464003b10 2023-12-16T21:58:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.097+0000 7fe476ffd640 1 -- 172.21.15.28:0/3889095397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4680a6a10 con 0x7fe464003b10 2023-12-16T21:58:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.097+0000 7fe47dee1640 1 -- 172.21.15.28:0/3889095397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464003b10 msgr2=0x7fe464003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.097+0000 7fe47dee1640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464003b10 0x7fe464003f10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe468009ae0 tx=0x7fe46801ca40 comp rx=0 tx=0).stop 2023-12-16T21:58:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- 172.21.15.28:0/3889095397 shutdown_connections 2023-12-16T21:58:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 --2- 172.21.15.28:0/3889095397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464003b10 0x7fe464003f10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- 172.21.15.28:0/3889095397 >> 172.21.15.28:0/3889095397 conn(0x7fe4780cf5d0 msgr2=0x7fe4780cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- 172.21.15.28:0/3889095397 shutdown_connections 2023-12-16T21:58:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- 172.21.15.28:0/3889095397 wait complete. 2023-12-16T21:58:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 Processor -- start 2023-12-16T21:58:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- start start 2023-12-16T21:58:37.111 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe464003b10 0x7fe47815e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe47815b320 0x7fe47814ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4780ce4a0 0x7fe4780ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47815c710 con 0x7fe4780ce4a0 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780cee60 con 0x7fe464003b10 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47815eea0 con 0x7fe47815b320 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4780ce4a0 0x7fe4780ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4780ce4a0 0x7fe4780ce920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48974/0 (socket says 172.21.15.28:48974) 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 -- 172.21.15.28:0/2789294915 learned_addr learned my addr 172.21.15.28:0/2789294915 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe4777fe640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe464003b10 0x7fe47815e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe46ffff640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe47815b320 0x7fe47814ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 -- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe47815b320 msgr2=0x7fe47814ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe47815b320 0x7fe47814ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 -- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe464003b10 msgr2=0x7fe47815e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe464003b10 0x7fe47815e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 -- 172.21.15.28:0/2789294915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47815f180 con 0x7fe4780ce4a0 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe4777fe640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe464003b10 0x7fe47815e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe477fff640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4780ce4a0 0x7fe4780ce920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe4780b3db0 tx=0x7fe47000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe474ff9640 1 -- 172.21.15.28:0/2789294915 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470006b60 con 0x7fe4780ce4a0 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe474ff9640 1 -- 172.21.15.28:0/2789294915 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe470006ce0 con 0x7fe4780ce4a0 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe474ff9640 1 -- 172.21.15.28:0/2789294915 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470003bc0 con 0x7fe4780ce4a0 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47815f460 con 0x7fe4780ce4a0 2023-12-16T21:58:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47815fa20 con 0x7fe4780ce4a0 2023-12-16T21:58:37.114 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.101+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4780674d0 con 0x7fe4780ce4a0 2023-12-16T21:58:37.114 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.105+0000 7fe474ff9640 1 -- 172.21.15.28:0/2789294915 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe470003d20 con 0x7fe4780ce4a0 2023-12-16T21:58:37.114 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.105+0000 7fe474ff9640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4580664a0 0x7fe458068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.114 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.105+0000 7fe474ff9640 1 -- 172.21.15.28:0/2789294915 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe4700831e0 con 0x7fe4780ce4a0 2023-12-16T21:58:37.114 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.105+0000 7fe4777fe640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4580664a0 0x7fe458068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.114 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.105+0000 7fe4777fe640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4580664a0 0x7fe458068960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe4680af000 tx=0x7fe4680046d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.114 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.109+0000 7fe474ff9640 1 -- 172.21.15.28:0/2789294915 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe470054440 con 0x7fe4780ce4a0 2023-12-16T21:58:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.317+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fe4780afaf0 con 0x7fe4780ce4a0 2023-12-16T21:58:37.328 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.321+0000 7fe474ff9640 1 -- 172.21.15.28:0/2789294915 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe470053fd0 con 0x7fe4780ce4a0 2023-12-16T21:58:37.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4580664a0 msgr2=0x7fe458068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4580664a0 0x7fe458068960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe4680af000 tx=0x7fe4680046d0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4780ce4a0 msgr2=0x7fe4780ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4780ce4a0 0x7fe4780ce920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe4780b3db0 tx=0x7fe47000eb90 comp rx=0 tx=0).stop 2023-12-16T21:58:37.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 shutdown_connections 2023-12-16T21:58:37.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe4580664a0 0x7fe458068960 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4780ce4a0 0x7fe4780ce920 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe47815b320 0x7fe47814ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 --2- 172.21.15.28:0/2789294915 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe464003b10 0x7fe47815e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 >> 172.21.15.28:0/2789294915 conn(0x7fe4780cf5d0 msgr2=0x7fe478160d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:37.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 shutdown_connections 2023-12-16T21:58:37.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.325+0000 7fe47dee1640 1 -- 172.21.15.28:0/2789294915 wait complete. 2023-12-16T21:58:37.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-12-16T21:58:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3df4d60640 1 Processor -- start 2023-12-16T21:58:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3df4d60640 1 -- start start 2023-12-16T21:58:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3df4d60640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00b4760 0x7f3df00d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3df4d60640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f3df00b4b40 0x7f3df00d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3df4d60640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df0143780 con 0x7f3df0142c00 2023-12-16T21:58:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3df4d60640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df0161dc0 con 0x7f3df00b4b40 2023-12-16T21:58:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3df4d60640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df015d7c0 con 0x7f3df00b4760 2023-12-16T21:58:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3deed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3df0142c00 0x7f3df0143070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57324/0 (socket says 172.21.15.28:57324) 2023-12-16T21:58:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3deed9d640 1 -- 172.21.15.28:0/4171369158 learned_addr learned my addr 172.21.15.28:0/4171369158 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dee59c640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00b4760 0x7f3df00d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138098938 0 0) 0x7f3df0143780 con 0x7f3df0142c00 2023-12-16T21:58:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3de7fff640 1 --2- 172.21.15.28:0/4171369158 >> v2:172.21.15.122:3300/0 conn(0x7f3df00b4b40 0x7f3df00d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ddc003530 con 0x7f3df0142c00 2023-12-16T21:58:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2792076554 0 0) 0x7f3ddc003530 con 0x7f3df0142c00 2023-12-16T21:58:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00b4760 msgr2=0x7f3df00d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00b4760 0x7f3df00d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 >> v2:172.21.15.122:3300/0 conn(0x7f3df00b4b40 msgr2=0x7f3df00d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 --2- 172.21.15.28:0/4171369158 >> v2:172.21.15.122:3300/0 conn(0x7f3df00b4b40 0x7f3df00d15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0140930 con 0x7f3df0142c00 2023-12-16T21:58:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f3de8001a00 con 0x7f3df0142c00 2023-12-16T21:58:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 >> v1:172.21.15.28:6789/0 conn(0x7f3df0142c00 legacy=0x7f3df0143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ddc003980 0x7f3ddc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00b4760 0x7f3ddc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ddc004d10 0x7f3ddc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0161dc0 con 0x7f3ddc004d10 2023-12-16T21:58:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df015d7c0 con 0x7f3ddc003980 2023-12-16T21:58:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc00b990 con 0x7f3df00b4760 2023-12-16T21:58:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3deed9d640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ddc004d10 0x7f3ddc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3deed9d640 1 -- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00b4760 msgr2=0x7f3ddc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3deed9d640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00b4760 0x7f3ddc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3deed9d640 1 -- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ddc003980 msgr2=0x7f3ddc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3deed9d640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ddc003980 0x7f3ddc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.453+0000 7f3deed9d640 1 -- 172.21.15.28:0/4171369158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ddc003530 con 0x7f3ddc004d10 2023-12-16T21:58:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3deed9d640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ddc004d10 0x7f3ddc005190 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3de800a840 tx=0x7f3de801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20da8ecd5640d341 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de80ad630 con 0x7f3ddc004d10 2023-12-16T21:58:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de80a8070 con 0x7f3ddc004d10 2023-12-16T21:58:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3dedd9b640 1 -- 172.21.15.28:0/4171369158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de80ad9f0 con 0x7f3ddc004d10 2023-12-16T21:58:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3df4d60640 1 -- 172.21.15.28:0/4171369158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ddc004d10 msgr2=0x7f3ddc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3df4d60640 1 --2- 172.21.15.28:0/4171369158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ddc004d10 0x7f3ddc005190 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3de800a840 tx=0x7f3de801d340 comp rx=0 tx=0).stop 2023-12-16T21:58:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3deed9d640 1 -- 172.21.15.28:0/4171369158 reap_dead start 2023-12-16T21:58:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3df4d60640 1 -- 172.21.15.28:0/4171369158 shutdown_connections 2023-12-16T21:58:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3df4d60640 1 -- 172.21.15.28:0/4171369158 >> 172.21.15.28:0/4171369158 conn(0x7f3df006a530 msgr2=0x7f3df006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3df4d60640 1 -- 172.21.15.28:0/4171369158 shutdown_connections 2023-12-16T21:58:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3df4d60640 1 -- 172.21.15.28:0/4171369158 wait complete. 2023-12-16T21:58:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3df4d60640 1 Processor -- start 2023-12-16T21:58:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.457+0000 7f3df4d60640 1 -- start start 2023-12-16T21:58:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3ddc004d10 0x7f3df0155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00cbe90 0x7f3df0160ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00cc310 0x7f3df00cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00cba70 con 0x7f3df00cc310 2023-12-16T21:58:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0155680 con 0x7f3df00cbe90 2023-12-16T21:58:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00cadf0 con 0x7f3ddc004d10 2023-12-16T21:58:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00cc310 0x7f3df00cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00cc310 0x7f3df00cdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36710/0 (socket says 172.21.15.28:36710) 2023-12-16T21:58:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 -- 172.21.15.28:0/2820410374 learned_addr learned my addr 172.21.15.28:0/2820410374 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3dee59c640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3ddc004d10 0x7f3df0155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3de7fff640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00cbe90 0x7f3df0160ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 -- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3ddc004d10 msgr2=0x7f3df0155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3ddc004d10 0x7f3df0155140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 -- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00cbe90 msgr2=0x7f3df0160ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00cbe90 0x7f3df0160ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 -- 172.21.15.28:0/2820410374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00cb0d0 con 0x7f3df00cc310 2023-12-16T21:58:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3deed9d640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00cc310 0x7f3df00cdb00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3de800a840 tx=0x7f3ddc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3de6ffd640 1 -- 172.21.15.28:0/2820410374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de80017c0 con 0x7f3df00cc310 2023-12-16T21:58:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3de6ffd640 1 -- 172.21.15.28:0/2820410374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de8001920 con 0x7f3df00cc310 2023-12-16T21:58:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df00cb3b0 con 0x7f3df00cc310 2023-12-16T21:58:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3de6ffd640 1 -- 172.21.15.28:0/2820410374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8004d70 con 0x7f3df00cc310 2023-12-16T21:58:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df013f9f0 con 0x7f3df00cc310 2023-12-16T21:58:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df00cbbb0 con 0x7f3df00cc310 2023-12-16T21:58:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3de6ffd640 1 -- 172.21.15.28:0/2820410374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f3de80aa030 con 0x7f3df00cc310 2023-12-16T21:58:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3de6ffd640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd0066470 0x7f3dd0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3de6ffd640 1 -- 172.21.15.28:0/2820410374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3de811dc30 con 0x7f3df00cc310 2023-12-16T21:58:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3dee59c640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd0066470 0x7f3dd0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.461+0000 7f3dee59c640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd0066470 0x7f3dd0068930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3de00047c0 tx=0x7f3de0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.465+0000 7f3de6ffd640 1 -- 172.21.15.28:0/2820410374 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3df00cbbb0 con 0x7f3df00cc310 2023-12-16T21:58:37.679 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.677+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f3df00cbbb0 con 0x7f3df00cc310 2023-12-16T21:58:37.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3de6ffd640 1 -- 172.21.15.28:0/2820410374 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3df00cbbb0 con 0x7f3df00cc310 2023-12-16T21:58:37.686 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd0066470 msgr2=0x7f3dd0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.686 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd0066470 0x7f3dd0068930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3de00047c0 tx=0x7f3de0008040 comp rx=0 tx=0).stop 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00cc310 msgr2=0x7f3df00cdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00cc310 0x7f3df00cdb00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3de800a840 tx=0x7f3ddc005800 comp rx=0 tx=0).stop 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 shutdown_connections 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd0066470 0x7f3dd0068930 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00cc310 0x7f3df00cdb00 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00cbe90 0x7f3df0160ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 --2- 172.21.15.28:0/2820410374 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3ddc004d10 0x7f3df0155140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 >> 172.21.15.28:0/2820410374 conn(0x7f3df006a530 msgr2=0x7f3df006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 shutdown_connections 2023-12-16T21:58:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.681+0000 7f3df4d60640 1 -- 172.21.15.28:0/2820410374 wait complete. 2023-12-16T21:58:37.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.809+0000 7f328a6d8640 1 Processor -- start 2023-12-16T21:58:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.809+0000 7f328a6d8640 1 -- start start 2023-12-16T21:58:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f328a6d8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 0x7f328406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f328a6d8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32840710e0 con 0x7f3284070b70 2023-12-16T21:58:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f328a6d8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32840713c0 con 0x7f3284072180 2023-12-16T21:58:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f328a6d8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328415f010 con 0x7f3284144840 2023-12-16T21:58:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3288c75640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3284070b70 0x7f328415b610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57328/0 (socket says 172.21.15.28:57328) 2023-12-16T21:58:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3288c75640 1 -- 172.21.15.28:0/4099262762 learned_addr learned my addr 172.21.15.28:0/4099262762 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588558133 0 0) 0x7f32840710e0 con 0x7f3284070b70 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3283fff640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 0x7f328406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f326c003530 con 0x7f3284070b70 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841443088 0 0) 0x7f326c003530 con 0x7f3284070b70 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 msgr2=0x7f328406c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 0x7f328406c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 >> v1:172.21.15.122:6789/0 conn(0x7f3284072180 legacy=0x7f3284070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840716a0 con 0x7f3284070b70 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f327c001a00 con 0x7f3284070b70 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 >> v1:172.21.15.28:6789/0 conn(0x7f3284070b70 legacy=0x7f328415b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326c0039b0 0x7f326c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 0x7f326c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c004d10 0x7f326c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c00ba40 con 0x7f326c0039b0 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c00bb80 con 0x7f326c004d10 2023-12-16T21:58:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c00bd40 con 0x7f3284144840 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3283fff640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326c0039b0 0x7f326c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f32837fe640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 0x7f326c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3283fff640 1 -- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 msgr2=0x7f326c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3283fff640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 0x7f326c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3283fff640 1 -- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c004d10 msgr2=0x7f326c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3288c75640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c004d10 0x7f326c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3283fff640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c004d10 0x7f326c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3283fff640 1 -- 172.21.15.28:0/4099262762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c003530 con 0x7f326c0039b0 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f32837fe640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3284144840 0x7f326c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:37.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.813+0000 7f3288c75640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c004d10 0x7f326c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3283fff640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326c0039b0 0x7f326c003dd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3274009ab0 tx=0x7f327401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398bc426a7c7669a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32740a55d0 con 0x7f326c0039b0 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32740a5730 con 0x7f326c0039b0 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3282ffd640 1 -- 172.21.15.28:0/4099262762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32740a5ae0 con 0x7f326c0039b0 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- 172.21.15.28:0/4099262762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326c0039b0 msgr2=0x7f326c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 --2- 172.21.15.28:0/4099262762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f326c0039b0 0x7f326c003dd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3274009ab0 tx=0x7f327401ca00 comp rx=0 tx=0).stop 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 -- 172.21.15.28:0/4099262762 reap_dead start 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- 172.21.15.28:0/4099262762 shutdown_connections 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- 172.21.15.28:0/4099262762 >> 172.21.15.28:0/4099262762 conn(0x7f328406a530 msgr2=0x7f328406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- 172.21.15.28:0/4099262762 shutdown_connections 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- 172.21.15.28:0/4099262762 wait complete. 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 Processor -- start 2023-12-16T21:58:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- start start 2023-12-16T21:58:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c0039b0 0x7f328415aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f32840a8e90 0x7f328415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32840a9310 0x7f328415f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3284130260 con 0x7f32840a9310 2023-12-16T21:58:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3284130540 con 0x7f326c0039b0 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3284130820 con 0x7f32840a8e90 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32840a9310 0x7f328415f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32840a9310 0x7f328415f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36732/0 (socket says 172.21.15.28:36732) 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 -- 172.21.15.28:0/3814566278 learned_addr learned my addr 172.21.15.28:0/3814566278 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3283fff640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c0039b0 0x7f328415aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f32837fe640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f32840a8e90 0x7f328415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 -- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f32840a8e90 msgr2=0x7f328415ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f32840a8e90 0x7f328415ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 -- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c0039b0 msgr2=0x7f328415aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c0039b0 0x7f328415aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 -- 172.21.15.28:0/3814566278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284130b00 con 0x7f32840a9310 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3288c75640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32840a9310 0x7f328415f4a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f328406e770 tx=0x7f327c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3280ff9640 1 -- 172.21.15.28:0/3814566278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c00bb80 con 0x7f32840a9310 2023-12-16T21:58:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3280ff9640 1 -- 172.21.15.28:0/3814566278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c00bce0 con 0x7f32840a9310 2023-12-16T21:58:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f3280ff9640 1 -- 172.21.15.28:0/3814566278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c0072b0 con 0x7f32840a9310 2023-12-16T21:58:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328415af90 con 0x7f32840a9310 2023-12-16T21:58:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.817+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328413f8f0 con 0x7f32840a9310 2023-12-16T21:58:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.821+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328413fae0 con 0x7f32840a9310 2023-12-16T21:58:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.821+0000 7f3280ff9640 1 -- 172.21.15.28:0/3814566278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f327c007410 con 0x7f32840a9310 2023-12-16T21:58:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.821+0000 7f3280ff9640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3270066470 0x7f3270068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:37.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.821+0000 7f3280ff9640 1 -- 172.21.15.28:0/3814566278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f327c083fd0 con 0x7f32840a9310 2023-12-16T21:58:37.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.821+0000 7f3283fff640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3270066470 0x7f3270068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:37.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.821+0000 7f3283fff640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3270066470 0x7f3270068930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f32740af000 tx=0x7f32740046d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.825+0000 7f3280ff9640 1 -- 172.21.15.28:0/3814566278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f327c051860 con 0x7f32840a9310 2023-12-16T21:58:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3284069440 con 0x7f32840a9310 2023-12-16T21:58:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f3280ff9640 1 -- 172.21.15.28:0/3814566278 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+12845 (secure 0 0 0) 0x7f327c054e80 con 0x7f32840a9310 2023-12-16T21:58:37.985 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:37.985 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:35.129778+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3270066470 msgr2=0x7f3270068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3270066470 0x7f3270068930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f32740af000 tx=0x7f32740046d0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32840a9310 msgr2=0x7f328415f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32840a9310 0x7f328415f4a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f328406e770 tx=0x7f327c00b4a0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 shutdown_connections 2023-12-16T21:58:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3270066470 0x7f3270068930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32840a9310 0x7f328415f4a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f32840a8e90 0x7f328415ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 --2- 172.21.15.28:0/3814566278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f326c0039b0 0x7f328415aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 >> 172.21.15.28:0/3814566278 conn(0x7f328406a530 msgr2=0x7f328415dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 shutdown_connections 2023-12-16T21:58:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:37.981+0000 7f328a6d8640 1 -- 172.21.15.28:0/3814566278 wait complete. 2023-12-16T21:58:38.003 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:58:38.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:58:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f21051b4640 1 Processor -- start 2023-12-16T21:58:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f21051b4640 1 -- start start 2023-12-16T21:58:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f21051b4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21001449d0 0x7f210006a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f21051b4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f210015f5b0 0x7f21000cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f21051b4640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21000cc6a0 con 0x7f21000cd630 2023-12-16T21:58:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f21051b4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2100144db0 con 0x7f210015f5b0 2023-12-16T21:58:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f21051b4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210006ac30 con 0x7f21001449d0 2023-12-16T21:58:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20ff59e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f21000cd630 0x7f2100071590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57338/0 (socket says 172.21.15.28:57338) 2023-12-16T21:58:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20ff59e640 1 -- 172.21.15.28:0/194993298 learned_addr learned my addr 172.21.15.28:0/194993298 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fed9d640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21001449d0 0x7f210006a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fe59c640 1 --2- 172.21.15.28:0/194993298 >> v2:172.21.15.122:3300/0 conn(0x7f210015f5b0 0x7f21000cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529630474 0 0) 0x7f21000cc6a0 con 0x7f21000cd630 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e8003530 con 0x7f21000cd630 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2689383875 0 0) 0x7f20e8003530 con 0x7f21000cd630 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21001449d0 msgr2=0x7f210006a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21001449d0 0x7f210006a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 >> v2:172.21.15.122:3300/0 conn(0x7f210015f5b0 msgr2=0x7f21000cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 --2- 172.21.15.28:0/194993298 >> v2:172.21.15.122:3300/0 conn(0x7f210015f5b0 0x7f21000cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210006af10 con 0x7f21000cd630 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f20f8001a00 con 0x7f21000cd630 2023-12-16T21:58:38.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 >> v1:172.21.15.28:6789/0 conn(0x7f21000cd630 legacy=0x7f2100071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21001449d0 0x7f20e8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e8003f00 0x7f20e80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e8004d10 0x7f20e8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2100144db0 con 0x7f20e8003f00 2023-12-16T21:58:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210006ac30 con 0x7f20e8004d10 2023-12-16T21:58:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e800b990 con 0x7f21001449d0 2023-12-16T21:58:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fe59c640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e8003f00 0x7f20e80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20ff59e640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e8004d10 0x7f20e8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fed9d640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21001449d0 0x7f20e8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fe59c640 1 -- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21001449d0 msgr2=0x7f20e8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fe59c640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f21001449d0 0x7f20e8003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fe59c640 1 -- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e8004d10 msgr2=0x7f20e8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fe59c640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e8004d10 0x7f20e8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.121+0000 7f20fe59c640 1 -- 172.21.15.28:0/194993298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e8003530 con 0x7f20e8003f00 2023-12-16T21:58:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f20fe59c640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e8003f00 0x7f20e80047d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009ae0 tx=0x7f20f401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61db8aaba16d84b2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f40a5570 con 0x7f20e8003f00 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f40a56d0 con 0x7f20e8003f00 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f20fdd9b640 1 -- 172.21.15.28:0/194993298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f40a5a80 con 0x7f20e8003f00 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f21051b4640 1 -- 172.21.15.28:0/194993298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e8003f00 msgr2=0x7f20e80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f21051b4640 1 --2- 172.21.15.28:0/194993298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e8003f00 0x7f20e80047d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009ae0 tx=0x7f20f401ca30 comp rx=0 tx=0).stop 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f20ff59e640 1 -- 172.21.15.28:0/194993298 reap_dead start 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f21051b4640 1 -- 172.21.15.28:0/194993298 shutdown_connections 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f21051b4640 1 -- 172.21.15.28:0/194993298 >> 172.21.15.28:0/194993298 conn(0x7f210006d510 msgr2=0x7f210006d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f21051b4640 1 -- 172.21.15.28:0/194993298 shutdown_connections 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f21051b4640 1 -- 172.21.15.28:0/194993298 wait complete. 2023-12-16T21:58:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f21051b4640 1 Processor -- start 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.125+0000 7f21051b4640 1 -- start start 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f20e8003f00 0x7f210015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21001f5420 0x7f21000c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21001f4720 0x7f21001f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21001f50f0 con 0x7f21001f4720 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000caf20 con 0x7f21001f5420 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21000cb200 con 0x7f20e8003f00 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21001f4720 0x7f21001f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21001f4720 0x7f21001f4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36744/0 (socket says 172.21.15.28:36744) 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 -- 172.21.15.28:0/2587951681 learned_addr learned my addr 172.21.15.28:0/2587951681 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20fed9d640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f20e8003f00 0x7f210015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20fe59c640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21001f5420 0x7f21000c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 -- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f20e8003f00 msgr2=0x7f210015a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f20e8003f00 0x7f210015a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 -- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21001f5420 msgr2=0x7f21000c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21001f5420 0x7f21000c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 -- 172.21.15.28:0/2587951681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21000cb4e0 con 0x7f21001f4720 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20ff59e640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21001f4720 0x7f21001f4bb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f20f8000f90 tx=0x7f20f8011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20e77fe640 1 -- 172.21.15.28:0/2587951681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f800f920 con 0x7f21001f4720 2023-12-16T21:58:38.138 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20e77fe640 1 -- 172.21.15.28:0/2587951681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f800fa80 con 0x7f21001f4720 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21000c8df0 con 0x7f21001f4720 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20e77fe640 1 -- 172.21.15.28:0/2587951681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f800fe30 con 0x7f21001f4720 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210015aa70 con 0x7f21001f4720 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21000674d0 con 0x7f21001f4720 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20e77fe640 1 -- 172.21.15.28:0/2587951681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f20f801f460 con 0x7f21001f4720 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20e77fe640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f20ec066470 0x7f20ec068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20e77fe640 1 -- 172.21.15.28:0/2587951681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f20f8086c10 con 0x7f21001f4720 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.129+0000 7f20fed9d640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f20ec066470 0x7f20ec068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.133+0000 7f20fed9d640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f20ec066470 0x7f20ec068930 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f20f0009950 tx=0x7f20f0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.133+0000 7f20e77fe640 1 -- 172.21.15.28:0/2587951681 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f20f80543e0 con 0x7f21001f4720 2023-12-16T21:58:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.285+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21000c92a0 con 0x7f21001f4720 2023-12-16T21:58:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.285+0000 7f20e77fe640 1 -- 172.21.15.28:0/2587951681 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v32) v1 ==== 93+0+10 (secure 0 0 0) 0x7f20f8057a00 con 0x7f21001f4720 2023-12-16T21:58:38.292 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-16T21:58:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f20ec066470 msgr2=0x7f20ec068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f20ec066470 0x7f20ec068930 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f20f0009950 tx=0x7f20f0008040 comp rx=0 tx=0).stop 2023-12-16T21:58:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21001f4720 msgr2=0x7f21001f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21001f4720 0x7f21001f4bb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f20f8000f90 tx=0x7f20f8011c20 comp rx=0 tx=0).stop 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 shutdown_connections 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f20ec066470 0x7f20ec068930 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21001f4720 0x7f21001f4bb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f21001f5420 0x7f21000c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 --2- 172.21.15.28:0/2587951681 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f20e8003f00 0x7f210015a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 >> 172.21.15.28:0/2587951681 conn(0x7f210006d510 msgr2=0x7f2100073460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 shutdown_connections 2023-12-16T21:58:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.289+0000 7f21051b4640 1 -- 172.21.15.28:0/2587951681 wait complete. 2023-12-16T21:58:38.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-16T21:58:38.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.417+0000 7f02609ff640 1 Processor -- start 2023-12-16T21:58:38.423 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.417+0000 7f02609ff640 1 -- start start 2023-12-16T21:58:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f02609ff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f025c06c9d0 0x7f025c070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f02609ff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c072180 0x7f025c0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f02609ff640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f025c0710e0 con 0x7f025c144840 2023-12-16T21:58:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f02609ff640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f025c15f010 con 0x7f025c06c9d0 2023-12-16T21:58:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f02609ff640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c070ee0 con 0x7f025c072180 2023-12-16T21:58:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f0259d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f025c06c9d0 0x7f025c070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.425 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f025a59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f025c144840 0x7f025c06c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57350/0 (socket says 172.21.15.28:57350) 2023-12-16T21:58:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f0259d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f025c06c9d0 0x7f025c070460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36756/0 (socket says 172.21.15.28:36756) 2023-12-16T21:58:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f025a59c640 1 -- 172.21.15.28:0/846745932 learned_addr learned my addr 172.21.15.28:0/846745932 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f025ad9d640 1 --2- 172.21.15.28:0/846745932 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c072180 0x7f025c0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f0259d9b640 1 -- 172.21.15.28:0/846745932 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c072180 msgr2=0x7f025c0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f0259d9b640 1 --2- 172.21.15.28:0/846745932 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c072180 0x7f025c0709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f0259d9b640 1 -- 172.21.15.28:0/846745932 >> v1:172.21.15.28:6789/0 conn(0x7f025c144840 legacy=0x7f025c06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f025959a640 1 -- 172.21.15.28:0/846745932 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069144915 0 0) 0x7f025c0710e0 con 0x7f025c144840 2023-12-16T21:58:38.426 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f0259d9b640 1 -- 172.21.15.28:0/846745932 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c0713c0 con 0x7f025c06c9d0 2023-12-16T21:58:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f0259d9b640 1 --2- 172.21.15.28:0/846745932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c06c9d0 0x7f025c070460 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f02500097c0 tx=0x7f025001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea786d3b66ced62 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f025959a640 1 -- 172.21.15.28:0/846745932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02500ab650 con 0x7f025c06c9d0 2023-12-16T21:58:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f025959a640 1 -- 172.21.15.28:0/846745932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02500ab7b0 con 0x7f025c06c9d0 2023-12-16T21:58:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.421+0000 7f025959a640 1 -- 172.21.15.28:0/846745932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02500abb40 con 0x7f025c06c9d0 2023-12-16T21:58:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- 172.21.15.28:0/846745932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c06c9d0 msgr2=0x7f025c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 --2- 172.21.15.28:0/846745932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c06c9d0 0x7f025c070460 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f02500097c0 tx=0x7f025001c450 comp rx=0 tx=0).stop 2023-12-16T21:58:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- 172.21.15.28:0/846745932 shutdown_connections 2023-12-16T21:58:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 --2- 172.21.15.28:0/846745932 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c072180 0x7f025c0709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 --2- 172.21.15.28:0/846745932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c06c9d0 0x7f025c070460 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- 172.21.15.28:0/846745932 >> 172.21.15.28:0/846745932 conn(0x7f025c06a530 msgr2=0x7f025c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:38.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- 172.21.15.28:0/846745932 shutdown_connections 2023-12-16T21:58:38.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- 172.21.15.28:0/846745932 wait complete. 2023-12-16T21:58:38.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 Processor -- start 2023-12-16T21:58:38.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- start start 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c06c9d0 0x7f025c15a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f025c072180 0x7f025c1606c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c144840 0x7f025c13c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c15aaa0 con 0x7f025c144840 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0c9480 con 0x7f025c072180 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f025c0c9760 con 0x7f025c06c9d0 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c144840 0x7f025c13c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c144840 0x7f025c13c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36762/0 (socket says 172.21.15.28:36762) 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 -- 172.21.15.28:0/2122072665 learned_addr learned my addr 172.21.15.28:0/2122072665 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 -- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c06c9d0 msgr2=0x7f025c15a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025a59c640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c06c9d0 0x7f025c15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f0259d9b640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f025c072180 0x7f025c1606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c06c9d0 0x7f025c15a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 -- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f025c072180 msgr2=0x7f025c1606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f025c072180 0x7f025c1606c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 -- 172.21.15.28:0/2122072665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0250009420 con 0x7f025c144840 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f025ad9d640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c144840 0x7f025c13c3a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0254007e40 tx=0x7f025400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f024affd640 1 -- 172.21.15.28:0/2122072665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254014620 con 0x7f025c144840 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f024affd640 1 -- 172.21.15.28:0/2122072665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025400ee90 con 0x7f025c144840 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f025c0b2260 con 0x7f025c144840 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f024affd640 1 -- 172.21.15.28:0/2122072665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02540149e0 con 0x7f025c144840 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.425+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f025c15aeb0 con 0x7f025c144840 2023-12-16T21:58:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.429+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025c0b20c0 con 0x7f025c144840 2023-12-16T21:58:38.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.429+0000 7f024affd640 1 -- 172.21.15.28:0/2122072665 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f0254014b40 con 0x7f025c144840 2023-12-16T21:58:38.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.429+0000 7f024affd640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f023c066470 0x7f023c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.429+0000 7f024affd640 1 -- 172.21.15.28:0/2122072665 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f02540844d0 con 0x7f025c144840 2023-12-16T21:58:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.429+0000 7f025a59c640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f023c066470 0x7f023c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.429+0000 7f025a59c640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f023c066470 0x7f023c068930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f024c000f70 tx=0x7f024c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.436 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.433+0000 7f024affd640 1 -- 172.21.15.28:0/2122072665 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f025c0b20c0 con 0x7f025c144840 2023-12-16T21:58:38.589 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.585+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f025c0b20c0 con 0x7f025c144840 2023-12-16T21:58:38.589 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.585+0000 7f024affd640 1 -- 172.21.15.28:0/2122072665 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v32) v1 ==== 84+0+65 (secure 0 0 0) 0x7f025c0b20c0 con 0x7f025c144840 2023-12-16T21:58:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2023-12-16T21:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2023-12-16T21:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2023-12-16T21:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2023-12-16T21:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:38.590 INFO:teuthology.orchestra.run.smithi028.stdout:] 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f023c066470 msgr2=0x7f023c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f023c066470 0x7f023c068930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f024c000f70 tx=0x7f024c009040 comp rx=0 tx=0).stop 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c144840 msgr2=0x7f025c13c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c144840 0x7f025c13c3a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0254007e40 tx=0x7f025400e4b0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 shutdown_connections 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f023c066470 0x7f023c068930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f025c144840 0x7f025c13c3a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f025c072180 0x7f025c1606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 --2- 172.21.15.28:0/2122072665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f025c06c9d0 0x7f025c15a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 >> 172.21.15.28:0/2122072665 conn(0x7f025c06a530 msgr2=0x7f025c06c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 shutdown_connections 2023-12-16T21:58:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.589+0000 7f02609ff640 1 -- 172.21.15.28:0/2122072665 wait complete. 2023-12-16T21:58:38.607 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-16T21:58:38.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-16T21:58:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69cdd5c640 1 Processor -- start 2023-12-16T21:58:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69cdd5c640 1 -- start start 2023-12-16T21:58:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69cdd5c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69c815a480 0x7f69c80ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69cdd5c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c806d6f0 0x7f69c806db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69cdd5c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c80ce370 con 0x7f69c80ae620 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69cdd5c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c80cd570 con 0x7f69c815a480 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69cdd5c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c806e0a0 con 0x7f69c806d6f0 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69c77fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69c815a480 0x7f69c80ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69c77fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69c815a480 0x7f69c80ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36768/0 (socket says 172.21.15.28:36768) 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69c77fe640 1 -- 172.21.15.28:0/3951662089 learned_addr learned my addr 172.21.15.28:0/3951662089 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.721+0000 7f69c7fff640 1 --2- 172.21.15.28:0/3951662089 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c806d6f0 0x7f69c806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69c77fe640 1 -- 172.21.15.28:0/3951662089 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c806d6f0 msgr2=0x7f69c806db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69c77fe640 1 --2- 172.21.15.28:0/3951662089 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c806d6f0 0x7f69c806db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69c77fe640 1 -- 172.21.15.28:0/3951662089 >> v1:172.21.15.122:6789/0 conn(0x7f69c80ae620 legacy=0x7f69c80cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69c77fe640 1 -- 172.21.15.28:0/3951662089 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c80cd760 con 0x7f69c815a480 2023-12-16T21:58:38.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69c77fe640 1 --2- 172.21.15.28:0/3951662089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c815a480 0x7f69c80ce800 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009e80 tx=0x7f69b801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6d8b13f1d7359b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69c67fc640 1 -- 172.21.15.28:0/3951662089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80a58b0 con 0x7f69c815a480 2023-12-16T21:58:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69c67fc640 1 -- 172.21.15.28:0/3951662089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b80a5a10 con 0x7f69c815a480 2023-12-16T21:58:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69c67fc640 1 -- 172.21.15.28:0/3951662089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80a5da0 con 0x7f69c815a480 2023-12-16T21:58:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 -- 172.21.15.28:0/3951662089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c815a480 msgr2=0x7f69c80ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/3951662089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c815a480 0x7f69c80ce800 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009e80 tx=0x7f69b801c9f0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 -- 172.21.15.28:0/3951662089 shutdown_connections 2023-12-16T21:58:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/3951662089 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c806d6f0 0x7f69c806db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/3951662089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c815a480 0x7f69c80ce800 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 -- 172.21.15.28:0/3951662089 >> 172.21.15.28:0/3951662089 conn(0x7f69c80b0c50 msgr2=0x7f69c80b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 -- 172.21.15.28:0/3951662089 shutdown_connections 2023-12-16T21:58:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 -- 172.21.15.28:0/3951662089 wait complete. 2023-12-16T21:58:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.725+0000 7f69cdd5c640 1 Processor -- start 2023-12-16T21:58:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 -- start start 2023-12-16T21:58:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c806d6f0 0x7f69c8150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69c80ae620 0x7f69c814ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c815a480 0x7f69c80c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c81512b0 con 0x7f69c806d6f0 2023-12-16T21:58:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c8159d10 con 0x7f69c80ae620 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c814cfc0 con 0x7f69c815a480 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c806d6f0 0x7f69c8150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c806d6f0 0x7f69c8150d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36778/0 (socket says 172.21.15.28:36778) 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 -- 172.21.15.28:0/223492539 learned_addr learned my addr 172.21.15.28:0/223492539 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c7fff640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c815a480 0x7f69c80c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c6ffd640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69c80ae620 0x7f69c814ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 -- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c815a480 msgr2=0x7f69c80c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c815a480 0x7f69c80c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 -- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69c80ae620 msgr2=0x7f69c814ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69c80ae620 0x7f69c814ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 -- 172.21.15.28:0/223492539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b8009b30 con 0x7f69c806d6f0 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c77fe640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c806d6f0 0x7f69c8150d70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69b80af040 tx=0x7f69b8019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69affff640 1 -- 172.21.15.28:0/223492539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80b8670 con 0x7f69c806d6f0 2023-12-16T21:58:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69affff640 1 -- 172.21.15.28:0/223492539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b80b87f0 con 0x7f69c806d6f0 2023-12-16T21:58:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c814d260 con 0x7f69c806d6f0 2023-12-16T21:58:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69affff640 1 -- 172.21.15.28:0/223492539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80b8ba0 con 0x7f69c806d6f0 2023-12-16T21:58:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c814d820 con 0x7f69c806d6f0 2023-12-16T21:58:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c804e850 con 0x7f69c806d6f0 2023-12-16T21:58:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69affff640 1 -- 172.21.15.28:0/223492539 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f69b80b8d00 con 0x7f69c806d6f0 2023-12-16T21:58:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69affff640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69b4066470 0x7f69b4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69affff640 1 -- 172.21.15.28:0/223492539 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f69b811f0a0 con 0x7f69c806d6f0 2023-12-16T21:58:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.729+0000 7f69c6ffd640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69b4066470 0x7f69b4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.733+0000 7f69c6ffd640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69b4066470 0x7f69b4068930 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f69c80afbf0 tx=0x7f69bc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.733+0000 7f69affff640 1 -- 172.21.15.28:0/223492539 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f69b80ec870 con 0x7f69c806d6f0 2023-12-16T21:58:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:38.885+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f69c80693d0 con 0x7f69c806d6f0 2023-12-16T21:58:39.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.465+0000 7f69affff640 1 -- 172.21.15.28:0/223492539 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v33) v1 ==== 138+0+0 (secure 0 0 0) 0x7f69b80efe90 con 0x7f69c806d6f0 2023-12-16T21:58:39.470 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2023-12-16T21:58:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69b4066470 msgr2=0x7f69b4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69b4066470 0x7f69b4068930 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f69c80afbf0 tx=0x7f69bc009040 comp rx=0 tx=0).stop 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c806d6f0 msgr2=0x7f69c8150d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c806d6f0 0x7f69c8150d70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69b80af040 tx=0x7f69b8019790 comp rx=0 tx=0).stop 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 shutdown_connections 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f69b4066470 0x7f69b4068930 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f69c815a480 0x7f69c80c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69c80ae620 0x7f69c814ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 --2- 172.21.15.28:0/223492539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69c806d6f0 0x7f69c8150d70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 >> 172.21.15.28:0/223492539 conn(0x7f69c80b0c50 msgr2=0x7f69c80aea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 shutdown_connections 2023-12-16T21:58:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.469+0000 7f69cdd5c640 1 -- 172.21.15.28:0/223492539 wait complete. 2023-12-16T21:58:39.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-16T21:58:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.625+0000 7f190cb60640 1 Processor -- start 2023-12-16T21:58:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.625+0000 7f190cb60640 1 -- start start 2023-12-16T21:58:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815ab70 0x7f190806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f190815b3b0 0x7f19080cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190806f610 con 0x7f190815af50 2023-12-16T21:58:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19080cefc0 con 0x7f190815b3b0 2023-12-16T21:58:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19080ce9b0 con 0x7f190815ab70 2023-12-16T21:58:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f1906d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f190815b3b0 0x7f19080cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f1906d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f190815b3b0 0x7f19080cf450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36788/0 (socket says 172.21.15.28:36788) 2023-12-16T21:58:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f1906d9d640 1 -- 172.21.15.28:0/4014529285 learned_addr learned my addr 172.21.15.28:0/4014529285 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190659c640 1 --2- 172.21.15.28:0/4014529285 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815ab70 0x7f190806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f1906d9d640 1 -- 172.21.15.28:0/4014529285 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815ab70 msgr2=0x7f190806ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f1906d9d640 1 --2- 172.21.15.28:0/4014529285 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815ab70 0x7f190806ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f1906d9d640 1 -- 172.21.15.28:0/4014529285 >> v1:172.21.15.122:6789/0 conn(0x7f190815af50 legacy=0x7f19080b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f1906d9d640 1 -- 172.21.15.28:0/4014529285 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190806f050 con 0x7f190815b3b0 2023-12-16T21:58:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f1906d9d640 1 --2- 172.21.15.28:0/4014529285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f19080cf450 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1900009e50 tx=0x7f190001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d754c3a49eabac0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190559a640 1 -- 172.21.15.28:0/4014529285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19000a56a0 con 0x7f190815b3b0 2023-12-16T21:58:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190559a640 1 -- 172.21.15.28:0/4014529285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19000a5800 con 0x7f190815b3b0 2023-12-16T21:58:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190559a640 1 -- 172.21.15.28:0/4014529285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19000a5b90 con 0x7f190815b3b0 2023-12-16T21:58:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 -- 172.21.15.28:0/4014529285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 msgr2=0x7f19080cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 --2- 172.21.15.28:0/4014529285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f19080cf450 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1900009e50 tx=0x7f190001c910 comp rx=0 tx=0).stop 2023-12-16T21:58:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 -- 172.21.15.28:0/4014529285 shutdown_connections 2023-12-16T21:58:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 --2- 172.21.15.28:0/4014529285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f19080cf450 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 --2- 172.21.15.28:0/4014529285 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815ab70 0x7f190806ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 -- 172.21.15.28:0/4014529285 >> 172.21.15.28:0/4014529285 conn(0x7f190806d510 msgr2=0x7f190806d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 -- 172.21.15.28:0/4014529285 shutdown_connections 2023-12-16T21:58:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.629+0000 7f190cb60640 1 -- 172.21.15.28:0/4014529285 wait complete. 2023-12-16T21:58:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 Processor -- start 2023-12-16T21:58:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 -- start start 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f190815ab70 0x7f190814c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815af50 0x7f19081599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f190815c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19081670b0 con 0x7f190815b3b0 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1908167390 con 0x7f190815ab70 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1908167670 con 0x7f190815af50 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f190815c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f190815c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36804/0 (socket says 172.21.15.28:36804) 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 -- 172.21.15.28:0/3932901733 learned_addr learned my addr 172.21.15.28:0/3932901733 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190659c640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f190815ab70 0x7f190814c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1905d9b640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815af50 0x7f19081599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 -- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815af50 msgr2=0x7f19081599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815af50 0x7f19081599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 -- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f190815ab70 msgr2=0x7f190814c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f190815ab70 0x7f190814c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 -- 172.21.15.28:0/3932901733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1900009b30 con 0x7f190815b3b0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f1906d9d640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f190815c810 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f19000af040 tx=0x7f19000196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f18f6ffd640 1 -- 172.21.15.28:0/3932901733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19000b8670 con 0x7f190815b3b0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f18f6ffd640 1 -- 172.21.15.28:0/3932901733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19000b87d0 con 0x7f190815b3b0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f18f6ffd640 1 -- 172.21.15.28:0/3932901733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19000b8b80 con 0x7f190815b3b0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1908167950 con 0x7f190815b3b0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190814cea0 con 0x7f190815b3b0 2023-12-16T21:58:39.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.633+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190814d030 con 0x7f190815b3b0 2023-12-16T21:58:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.637+0000 7f18f6ffd640 1 -- 172.21.15.28:0/3932901733 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f19000b8ce0 con 0x7f190815b3b0 2023-12-16T21:58:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.637+0000 7f18f6ffd640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18e8066470 0x7f18e8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.637+0000 7f18f6ffd640 1 -- 172.21.15.28:0/3932901733 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f190011f120 con 0x7f190815b3b0 2023-12-16T21:58:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.637+0000 7f190659c640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18e8066470 0x7f18e8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.637+0000 7f190659c640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18e8066470 0x7f18e8068930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f18f800b9d0 tx=0x7f18f8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.641+0000 7f18f6ffd640 1 -- 172.21.15.28:0/3932901733 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f19000ec8f0 con 0x7f190815b3b0 2023-12-16T21:58:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:39.797+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f190806f610 con 0x7f190815b3b0 2023-12-16T21:58:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.469+0000 7f18f6ffd640 1 -- 172.21.15.28:0/3932901733 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v34) v1 ==== 170+0+0 (secure 0 0 0) 0x7f190806f610 con 0x7f190815b3b0 2023-12-16T21:58:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18e8066470 msgr2=0x7f18e8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18e8066470 0x7f18e8068930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f18f800b9d0 tx=0x7f18f8008040 comp rx=0 tx=0).stop 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 msgr2=0x7f190815c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f190815c810 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f19000af040 tx=0x7f19000196b0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 shutdown_connections 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f18e8066470 0x7f18e8068930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f190815b3b0 0x7f190815c810 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f190815af50 0x7f19081599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 --2- 172.21.15.28:0/3932901733 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f190815ab70 0x7f190814c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 >> 172.21.15.28:0/3932901733 conn(0x7f190806d510 msgr2=0x7f1908071590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 shutdown_connections 2023-12-16T21:58:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.473+0000 7f190cb60640 1 -- 172.21.15.28:0/3932901733 wait complete. 2023-12-16T21:58:40.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-16T21:58:40.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.629+0000 7f6c4b333640 1 Processor -- start 2023-12-16T21:58:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.629+0000 7f6c4b333640 1 -- start start 2023-12-16T21:58:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c4b333640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c44071d90 0x7f6c44072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c4b333640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c4415adb0 0x7f6c4413d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c4b333640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4413d2b0 con 0x7f6c4413f5f0 2023-12-16T21:58:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c4b333640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c44140520 con 0x7f6c4415adb0 2023-12-16T21:58:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c4b333640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440cd7f0 con 0x7f6c44071d90 2023-12-16T21:58:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c488ce640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c4415adb0 0x7f6c4413d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c488ce640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c4415adb0 0x7f6c4413d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36806/0 (socket says 172.21.15.28:36806) 2023-12-16T21:58:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c488ce640 1 -- 172.21.15.28:0/493400330 learned_addr learned my addr 172.21.15.28:0/493400330 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c490cf640 1 --2- 172.21.15.28:0/493400330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c44071d90 0x7f6c44072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c488ce640 1 -- 172.21.15.28:0/493400330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c44071d90 msgr2=0x7f6c44072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c488ce640 1 --2- 172.21.15.28:0/493400330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c44071d90 0x7f6c44072170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c488ce640 1 -- 172.21.15.28:0/493400330 >> v1:172.21.15.28:6789/0 conn(0x7f6c4413f5f0 legacy=0x7f6c440ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c488ce640 1 -- 172.21.15.28:0/493400330 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c440cdd70 con 0x7f6c4415adb0 2023-12-16T21:58:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c490cf640 1 --2- 172.21.15.28:0/493400330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c44071d90 0x7f6c44072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c488ce640 1 --2- 172.21.15.28:0/493400330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c4415adb0 0x7f6c4413d560 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400bb10 tx=0x7f6c3401ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e62dd90c0094e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.633+0000 7f6c3bfff640 1 -- 172.21.15.28:0/493400330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340a7be0 con 0x7f6c4415adb0 2023-12-16T21:58:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c3bfff640 1 -- 172.21.15.28:0/493400330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c340a7d40 con 0x7f6c4415adb0 2023-12-16T21:58:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c3bfff640 1 -- 172.21.15.28:0/493400330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c340af660 con 0x7f6c4415adb0 2023-12-16T21:58:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- 172.21.15.28:0/493400330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c4415adb0 msgr2=0x7f6c4413d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 --2- 172.21.15.28:0/493400330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c4415adb0 0x7f6c4413d560 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400bb10 tx=0x7f6c3401ec10 comp rx=0 tx=0).stop 2023-12-16T21:58:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- 172.21.15.28:0/493400330 shutdown_connections 2023-12-16T21:58:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 --2- 172.21.15.28:0/493400330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c4415adb0 0x7f6c4413d560 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 --2- 172.21.15.28:0/493400330 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c44071d90 0x7f6c44072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- 172.21.15.28:0/493400330 >> 172.21.15.28:0/493400330 conn(0x7f6c440b0cc0 msgr2=0x7f6c440b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- 172.21.15.28:0/493400330 shutdown_connections 2023-12-16T21:58:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- 172.21.15.28:0/493400330 wait complete. 2023-12-16T21:58:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 Processor -- start 2023-12-16T21:58:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- start start 2023-12-16T21:58:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c44071d90 0x7f6c440ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4413f5f0 0x7f6c440ab700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c4415adb0 0x7f6c44141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4413b0b0 con 0x7f6c44071d90 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4413a490 con 0x7f6c4413f5f0 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c44142740 con 0x7f6c4415adb0 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c44071d90 0x7f6c440ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c44071d90 0x7f6c440ab1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36820/0 (socket says 172.21.15.28:36820) 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 -- 172.21.15.28:0/3837883532 learned_addr learned my addr 172.21.15.28:0/3837883532 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c498d0640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c4415adb0 0x7f6c44141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c488ce640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4413f5f0 0x7f6c440ab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 -- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c4415adb0 msgr2=0x7f6c44141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c4415adb0 0x7f6c44141590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 -- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4413f5f0 msgr2=0x7f6c440ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4413f5f0 0x7f6c440ab700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 -- 172.21.15.28:0/3837883532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3400b770 con 0x7f6c44071d90 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c490cf640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c44071d90 0x7f6c440ab1c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000c820 tx=0x7f6c4000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c39ffb640 1 -- 172.21.15.28:0/3837883532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40007a30 con 0x7f6c44071d90 2023-12-16T21:58:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c39ffb640 1 -- 172.21.15.28:0/3837883532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c40007b90 con 0x7f6c44071d90 2023-12-16T21:58:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4413a780 con 0x7f6c44071d90 2023-12-16T21:58:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c39ffb640 1 -- 172.21.15.28:0/3837883532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4000ce10 con 0x7f6c44071d90 2023-12-16T21:58:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.637+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4413ad40 con 0x7f6c44071d90 2023-12-16T21:58:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.641+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c440693d0 con 0x7f6c44071d90 2023-12-16T21:58:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.641+0000 7f6c39ffb640 1 -- 172.21.15.28:0/3837883532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6c400113f0 con 0x7f6c44071d90 2023-12-16T21:58:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.641+0000 7f6c39ffb640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c28066470 0x7f6c28068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.641+0000 7f6c39ffb640 1 -- 172.21.15.28:0/3837883532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6c40083fb0 con 0x7f6c44071d90 2023-12-16T21:58:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.641+0000 7f6c488ce640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c28066470 0x7f6c28068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.641+0000 7f6c488ce640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c28066470 0x7f6c28068930 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6c340b1000 tx=0x7f6c340a9070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.645+0000 7f6c39ffb640 1 -- 172.21.15.28:0/3837883532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6c400b5290 con 0x7f6c44071d90 2023-12-16T21:58:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:40.833+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6c4406e710 con 0x7f6c44071d90 2023-12-16T21:58:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.493+0000 7f6c39ffb640 1 -- 172.21.15.28:0/3837883532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6c40054f20 con 0x7f6c44071d90 2023-12-16T21:58:41.498 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 4 and data pool 5 2023-12-16T21:58:41.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c28066470 msgr2=0x7f6c28068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c28066470 0x7f6c28068930 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6c340b1000 tx=0x7f6c340a9070 comp rx=0 tx=0).stop 2023-12-16T21:58:41.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c44071d90 msgr2=0x7f6c440ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c44071d90 0x7f6c440ab1c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6c4000c820 tx=0x7f6c4000cce0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 shutdown_connections 2023-12-16T21:58:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6c28066470 0x7f6c28068930 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6c4415adb0 0x7f6c44141590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4413f5f0 0x7f6c440ab700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 --2- 172.21.15.28:0/3837883532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c44071d90 0x7f6c440ab1c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 >> 172.21.15.28:0/3837883532 conn(0x7f6c440b0cc0 msgr2=0x7f6c440ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 shutdown_connections 2023-12-16T21:58:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.497+0000 7f6c4b333640 1 -- 172.21.15.28:0/3837883532 wait complete. 2023-12-16T21:58:41.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df6b1b640 1 Processor -- start 2023-12-16T21:58:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df6b1b640 1 -- start start 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df6b1b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df00d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df6b1b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3df00b3c70 0x7f3df015f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df6b1b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df00d3880 con 0x7f3df00b35b0 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df6b1b640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df00d4090 con 0x7f3df00b3c70 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df6b1b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df00cdc80 con 0x7f3df00cfe90 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df50b8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3df00b3c70 0x7f3df015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df50b8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3df00b3c70 0x7f3df015f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36822/0 (socket says 172.21.15.28:36822) 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df50b8640 1 -- 172.21.15.28:0/1003009314 learned_addr learned my addr 172.21.15.28:0/1003009314 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3deffff640 1 --2- 172.21.15.28:0/1003009314 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df00d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df50b8640 1 -- 172.21.15.28:0/1003009314 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 msgr2=0x7f3df00d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df50b8640 1 --2- 172.21.15.28:0/1003009314 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df00d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df50b8640 1 -- 172.21.15.28:0/1003009314 >> v1:172.21.15.122:6789/0 conn(0x7f3df00b35b0 legacy=0x7f3df00cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df50b8640 1 -- 172.21.15.28:0/1003009314 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00d3ad0 con 0x7f3df00b3c70 2023-12-16T21:58:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3deffff640 1 --2- 172.21.15.28:0/1003009314 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df00d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.657+0000 7f3df50b8640 1 --2- 172.21.15.28:0/1003009314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df015f060 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009e50 tx=0x7f3de801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250804974eab41ee server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3def7fe640 1 -- 172.21.15.28:0/1003009314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de80a57c0 con 0x7f3df00b3c70 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3def7fe640 1 -- 172.21.15.28:0/1003009314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de80a5920 con 0x7f3df00b3c70 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3def7fe640 1 -- 172.21.15.28:0/1003009314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de80a5cb0 con 0x7f3df00b3c70 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1003009314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 msgr2=0x7f3df015f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1003009314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df015f060 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009e50 tx=0x7f3de801c940 comp rx=0 tx=0).stop 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1003009314 shutdown_connections 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1003009314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df015f060 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1003009314 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df00d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1003009314 >> 172.21.15.28:0/1003009314 conn(0x7f3df009c4b0 msgr2=0x7f3df00d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1003009314 shutdown_connections 2023-12-16T21:58:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1003009314 wait complete. 2023-12-16T21:58:41.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 Processor -- start 2023-12-16T21:58:41.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- start start 2023-12-16T21:58:41.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00b35b0 0x7f3df014f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df0154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df0154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0155a80 con 0x7f3df00b3c70 2023-12-16T21:58:41.668 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0152820 con 0x7f3df00b35b0 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0155ea0 con 0x7f3df00cfe90 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df0154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df0154870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36830/0 (socket says 172.21.15.28:36830) 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 -- 172.21.15.28:0/1294445786 learned_addr learned my addr 172.21.15.28:0/1294445786 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df50b8640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df0154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df48b7640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00b35b0 0x7f3df014f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 -- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 msgr2=0x7f3df0154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df0154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 -- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00b35b0 msgr2=0x7f3df014f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00b35b0 0x7f3df014f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 -- 172.21.15.28:0/1294445786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8009b30 con 0x7f3df00b3c70 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3deffff640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df0154870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00c820 tx=0x7f3ddc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3ded7fa640 1 -- 172.21.15.28:0/1294445786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc0079d0 con 0x7f3df00b3c70 2023-12-16T21:58:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3ded7fa640 1 -- 172.21.15.28:0/1294445786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ddc007b30 con 0x7f3df00b3c70 2023-12-16T21:58:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3ded7fa640 1 -- 172.21.15.28:0/1294445786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc00ce10 con 0x7f3df00b3c70 2023-12-16T21:58:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df01552f0 con 0x7f3df00b3c70 2023-12-16T21:58:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.661+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df0152260 con 0x7f3df00b3c70 2023-12-16T21:58:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.665+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df01557e0 con 0x7f3df00b3c70 2023-12-16T21:58:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.665+0000 7f3ded7fa640 1 -- 172.21.15.28:0/1294445786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f3ddc0113f0 con 0x7f3df00b3c70 2023-12-16T21:58:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.665+0000 7f3ded7fa640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd4066470 0x7f3dd4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.665+0000 7f3ded7fa640 1 -- 172.21.15.28:0/1294445786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ddc083f30 con 0x7f3df00b3c70 2023-12-16T21:58:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.665+0000 7f3df48b7640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd4066470 0x7f3dd4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.665+0000 7f3df48b7640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd4066470 0x7f3dd4068930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3df015e860 tx=0x7f3de4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:41.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.669+0000 7f3ded7fa640 1 -- 172.21.15.28:0/1294445786 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3df01557e0 con 0x7f3df00b3c70 2023-12-16T21:58:41.830 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.825+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3df0155bc0 con 0x7f3df00b3c70 2023-12-16T21:58:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3ded7fa640 1 -- 172.21.15.28:0/1294445786 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f3df0155bc0 con 0x7f3df00b3c70 2023-12-16T21:58:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:41.833 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:40.841188+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:58:38.932657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:58:39.841920+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd4066470 msgr2=0x7f3dd4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd4066470 0x7f3dd4068930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3df015e860 tx=0x7f3de4009040 comp rx=0 tx=0).stop 2023-12-16T21:58:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 msgr2=0x7f3df0154870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df0154870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00c820 tx=0x7f3ddc00cce0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 shutdown_connections 2023-12-16T21:58:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3dd4066470 0x7f3dd4068930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3df00cfe90 0x7f3df0154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df00b3c70 0x7f3df0154870 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 --2- 172.21.15.28:0/1294445786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00b35b0 0x7f3df014f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 >> 172.21.15.28:0/1294445786 conn(0x7f3df009c4b0 msgr2=0x7f3df00d20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 shutdown_connections 2023-12-16T21:58:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.829+0000 7f3df6b1b640 1 -- 172.21.15.28:0/1294445786 wait complete. 2023-12-16T21:58:41.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f88306e8640 1 Processor -- start 2023-12-16T21:58:41.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f88306e8640 1 -- start start 2023-12-16T21:58:41.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f88306e8640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f882815d380 0x7f88280c9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f88306e8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 0x7f88280cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f88306e8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88280afe00 con 0x7f88280cc740 2023-12-16T21:58:41.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f88306e8640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88280af840 con 0x7f882815d380 2023-12-16T21:58:41.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f88306e8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88280afb20 con 0x7f88280cfab0 2023-12-16T21:58:41.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882ec85640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f88280cc740 0x7f88280c8470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57362/0 (socket says 172.21.15.28:57362) 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882ec85640 1 -- 172.21.15.28:0/1440916925 learned_addr learned my addr 172.21.15.28:0/1440916925 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882e484640 1 --2- 172.21.15.28:0/1440916925 >> v2:172.21.15.122:3300/0 conn(0x7f882815d380 0x7f88280c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880483794 0 0) 0x7f88280afe00 con 0x7f88280cc740 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8814003530 con 0x7f88280cc740 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882dc83640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 0x7f88280cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 870645004 0 0) 0x7f8814003530 con 0x7f88280cc740 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 msgr2=0x7f88280cc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 0x7f88280cc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 >> v2:172.21.15.122:3300/0 conn(0x7f882815d380 msgr2=0x7f88280c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 --2- 172.21.15.28:0/1440916925 >> v2:172.21.15.122:3300/0 conn(0x7f882815d380 0x7f88280c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280cbce0 con 0x7f88280cc740 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f8820001680 con 0x7f88280cc740 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 >> v1:172.21.15.28:6789/0 conn(0x7f88280cc740 legacy=0x7f88280c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 0x7f88140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8814004d10 0x7f8814005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280af840 con 0x7f8814004d10 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280afb20 con 0x7f8814003980 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881400b990 con 0x7f88280cfab0 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882ec85640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8814004d10 0x7f8814005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882e484640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882ec85640 1 -- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 msgr2=0x7f88140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882dc83640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 0x7f88140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882ec85640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 0x7f88140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882ec85640 1 -- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8814003980 msgr2=0x7f8814003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882ec85640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882ec85640 1 -- 172.21.15.28:0/1440916925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814003530 con 0x7f8814004d10 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882dc83640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88280cfab0 0x7f88140047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:41.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.973+0000 7f882e484640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8814003980 0x7f8814003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f882ec85640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8814004d10 0x7f8814005190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f882000c240 tx=0x7f882001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8d45754b679a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88200a5960 con 0x7f8814004d10 2023-12-16T21:58:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88200a5ac0 con 0x7f8814004d10 2023-12-16T21:58:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f882d482640 1 -- 172.21.15.28:0/1440916925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88200a5e90 con 0x7f8814004d10 2023-12-16T21:58:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 -- 172.21.15.28:0/1440916925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8814004d10 msgr2=0x7f8814005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 --2- 172.21.15.28:0/1440916925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8814004d10 0x7f8814005190 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f882000c240 tx=0x7f882001cc60 comp rx=0 tx=0).stop 2023-12-16T21:58:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f882ec85640 1 -- 172.21.15.28:0/1440916925 reap_dead start 2023-12-16T21:58:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 -- 172.21.15.28:0/1440916925 shutdown_connections 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 -- 172.21.15.28:0/1440916925 >> 172.21.15.28:0/1440916925 conn(0x7f8828071c50 msgr2=0x7f8828072030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 -- 172.21.15.28:0/1440916925 shutdown_connections 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 -- 172.21.15.28:0/1440916925 wait complete. 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 Processor -- start 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 -- start start 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88281f2250 0x7f88281f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f882815d8e0 0x7f88280cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88280cdea0 0x7f882815ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.977+0000 7f88306e8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280cd1f0 con 0x7f88280cdea0 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f88306e8640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882815e0f0 con 0x7f882815d8e0 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f88306e8640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882815e3d0 con 0x7f88281f2250 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88280cdea0 0x7f882815ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88280cdea0 0x7f882815ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36858/0 (socket says 172.21.15.28:36858) 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 -- 172.21.15.28:0/311418323 learned_addr learned my addr 172.21.15.28:0/311418323 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882e484640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88281f2250 0x7f88281f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882dc83640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f882815d8e0 0x7f88280cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 -- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88281f2250 msgr2=0x7f88281f6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88281f2250 0x7f88281f6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 -- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f882815d8e0 msgr2=0x7f88280cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f882815d8e0 0x7f88280cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 -- 172.21.15.28:0/311418323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882815e6b0 con 0x7f88280cdea0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882e484640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88281f2250 0x7f88281f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882dc83640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f882815d8e0 0x7f88280cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882ec85640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88280cdea0 0x7f882815ec20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f882000c240 tx=0x7f8814005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f881effd640 1 -- 172.21.15.28:0/311418323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88200a5960 con 0x7f88280cdea0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f881effd640 1 -- 172.21.15.28:0/311418323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88200072d0 con 0x7f88280cdea0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88280cd440 con 0x7f88280cdea0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f881effd640 1 -- 172.21.15.28:0/311418323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820007640 con 0x7f88280cdea0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882815f160 con 0x7f88280cdea0 2023-12-16T21:58:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882815e840 con 0x7f88280cdea0 2023-12-16T21:58:41.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f881effd640 1 -- 172.21.15.28:0/311418323 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f88200ae430 con 0x7f88280cdea0 2023-12-16T21:58:41.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f881effd640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:41.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f881effd640 1 -- 172.21.15.28:0/311418323 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88200b4070 con 0x7f88280cdea0 2023-12-16T21:58:41.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882e484640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:41.988 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.981+0000 7f882e484640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f880c0664a0 0x7f880c068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8824007380 tx=0x7f882400b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:41.989+0000 7f881effd640 1 -- 172.21.15.28:0/311418323 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f88200ead40 con 0x7f88280cdea0 2023-12-16T21:58:42.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.145+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8828069440 con 0x7f88280cdea0 2023-12-16T21:58:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f881effd640 1 -- 172.21.15.28:0/311418323 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f8820001c60 con 0x7f88280cdea0 2023-12-16T21:58:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:42.153 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:40.841188+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:58:38.932657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:58:39.841920+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:42.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f880c0664a0 msgr2=0x7f880c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f880c0664a0 0x7f880c068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8824007380 tx=0x7f882400b040 comp rx=0 tx=0).stop 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88280cdea0 msgr2=0x7f882815ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88280cdea0 0x7f882815ec20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f882000c240 tx=0x7f8814005800 comp rx=0 tx=0).stop 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 shutdown_connections 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88280cdea0 0x7f882815ec20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f882815d8e0 0x7f88280cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 --2- 172.21.15.28:0/311418323 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f88281f2250 0x7f88281f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 >> 172.21.15.28:0/311418323 conn(0x7f8828071c50 msgr2=0x7f8828160c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.149+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 shutdown_connections 2023-12-16T21:58:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.153+0000 7f88306e8640 1 -- 172.21.15.28:0/311418323 wait complete. 2023-12-16T21:58:42.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-16T21:58:42.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e6d92640 1 Processor -- start 2023-12-16T21:58:42.304 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e6d92640 1 -- start start 2023-12-16T21:58:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e6d92640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e0071590 0x7f00e0071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e6d92640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00e00b2fd0 0x7f00e00b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e6d92640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e006ff20 con 0x7f00e0143b30 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e6d92640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00e0071f50 con 0x7f00e00b2fd0 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e6d92640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e0072a70 con 0x7f00e0071590 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e532f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00e00b2fd0 0x7f00e00b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e532f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00e00b2fd0 0x7f00e00b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36866/0 (socket says 172.21.15.28:36866) 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e532f640 1 -- 172.21.15.28:0/976846474 learned_addr learned my addr 172.21.15.28:0/976846474 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00dffff640 1 --2- 172.21.15.28:0/976846474 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e0071590 0x7f00e0071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e532f640 1 -- 172.21.15.28:0/976846474 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e0071590 msgr2=0x7f00e0071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e532f640 1 --2- 172.21.15.28:0/976846474 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e0071590 0x7f00e0071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e532f640 1 -- 172.21.15.28:0/976846474 >> v1:172.21.15.122:6789/0 conn(0x7f00e0143b30 legacy=0x7f00e0072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.301+0000 7f00e532f640 1 -- 172.21.15.28:0/976846474 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e00cec90 con 0x7f00e00b2fd0 2023-12-16T21:58:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e532f640 1 --2- 172.21.15.28:0/976846474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e00b2fd0 0x7f00e00b3450 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009e50 tx=0x7f00d801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438cf0b625ec027f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00df7fe640 1 -- 172.21.15.28:0/976846474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d80a58a0 con 0x7f00e00b2fd0 2023-12-16T21:58:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00df7fe640 1 -- 172.21.15.28:0/976846474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d80a5a00 con 0x7f00e00b2fd0 2023-12-16T21:58:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00df7fe640 1 -- 172.21.15.28:0/976846474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d80a5d90 con 0x7f00e00b2fd0 2023-12-16T21:58:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- 172.21.15.28:0/976846474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e00b2fd0 msgr2=0x7f00e00b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 --2- 172.21.15.28:0/976846474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e00b2fd0 0x7f00e00b3450 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f00d8009e50 tx=0x7f00d801c9e0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- 172.21.15.28:0/976846474 shutdown_connections 2023-12-16T21:58:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 --2- 172.21.15.28:0/976846474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e00b2fd0 0x7f00e00b3450 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 --2- 172.21.15.28:0/976846474 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e0071590 0x7f00e0071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- 172.21.15.28:0/976846474 >> 172.21.15.28:0/976846474 conn(0x7f00e006d510 msgr2=0x7f00e006d920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- 172.21.15.28:0/976846474 shutdown_connections 2023-12-16T21:58:42.310 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- 172.21.15.28:0/976846474 wait complete. 2023-12-16T21:58:42.310 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 Processor -- start 2023-12-16T21:58:42.310 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- start start 2023-12-16T21:58:42.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00e0071590 0x7f00e0157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e00b2fd0 0x7f00e013f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.312 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e0143b30 0x7f00e013fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e0158130 con 0x7f00e0143b30 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e015a8e0 con 0x7f00e0071590 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e6d92640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e0163050 con 0x7f00e00b2fd0 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e532f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e0143b30 0x7f00e013fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e532f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e0143b30 0x7f00e013fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36882/0 (socket says 172.21.15.28:36882) 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.305+0000 7f00e532f640 1 -- 172.21.15.28:0/3294010688 learned_addr learned my addr 172.21.15.28:0/3294010688 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e4b2e640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00e0071590 0x7f00e0157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00dffff640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e00b2fd0 0x7f00e013f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e532f640 1 -- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e00b2fd0 msgr2=0x7f00e013f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e532f640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e00b2fd0 0x7f00e013f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e532f640 1 -- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00e0071590 msgr2=0x7f00e0157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e532f640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00e0071590 0x7f00e0157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e532f640 1 -- 172.21.15.28:0/3294010688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d8009b30 con 0x7f00e0143b30 2023-12-16T21:58:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e532f640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e0143b30 0x7f00e013fec0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f00d80af040 tx=0x7f00d8019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00dd7fa640 1 -- 172.21.15.28:0/3294010688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d80b8670 con 0x7f00e0143b30 2023-12-16T21:58:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00dd7fa640 1 -- 172.21.15.28:0/3294010688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d80b8810 con 0x7f00e0143b30 2023-12-16T21:58:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00dd7fa640 1 -- 172.21.15.28:0/3294010688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d80b8bc0 con 0x7f00e0143b30 2023-12-16T21:58:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e0158290 con 0x7f00e0143b30 2023-12-16T21:58:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e015cf40 con 0x7f00e0143b30 2023-12-16T21:58:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e00b33b0 con 0x7f00e0143b30 2023-12-16T21:58:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00dd7fa640 1 -- 172.21.15.28:0/3294010688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f00d80b8d20 con 0x7f00e0143b30 2023-12-16T21:58:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00dd7fa640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00c4066470 0x7f00c4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00dd7fa640 1 -- 172.21.15.28:0/3294010688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00d811f540 con 0x7f00e0143b30 2023-12-16T21:58:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.309+0000 7f00e4b2e640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00c4066470 0x7f00c4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.313+0000 7f00e4b2e640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00c4066470 0x7f00c4068930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f00d4000f70 tx=0x7f00d4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:42.319 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.313+0000 7f00dd7fa640 1 -- 172.21.15.28:0/3294010688 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f00d80eca40 con 0x7f00e0143b30 2023-12-16T21:58:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.505+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f00e00674d0 con 0x7f00e0143b30 2023-12-16T21:58:42.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00dd7fa640 1 -- 172.21.15.28:0/3294010688 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7f00d80f0060 con 0x7f00e0143b30 2023-12-16T21:58:42.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00c4066470 msgr2=0x7f00c4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00c4066470 0x7f00c4068930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f00d4000f70 tx=0x7f00d4009040 comp rx=0 tx=0).stop 2023-12-16T21:58:42.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e0143b30 msgr2=0x7f00e013fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e0143b30 0x7f00e013fec0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f00d80af040 tx=0x7f00d8019780 comp rx=0 tx=0).stop 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 shutdown_connections 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f00c4066470 0x7f00c4068930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00e0143b30 0x7f00e013fec0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f00e00b2fd0 0x7f00e013f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 --2- 172.21.15.28:0/3294010688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00e0071590 0x7f00e0157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 >> 172.21.15.28:0/3294010688 conn(0x7f00e006d510 msgr2=0x7f00e006d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 shutdown_connections 2023-12-16T21:58:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.529+0000 7f00e6d92640 1 -- 172.21.15.28:0/3294010688 wait complete. 2023-12-16T21:58:42.553 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-12-16T21:58:42.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-16T21:58:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f107cc640 1 Processor -- start 2023-12-16T21:58:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f107cc640 1 -- start start 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f107cc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5f08071d90 0x7f5f08072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f107cc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f0815adb0 0x7f5f0813d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f107cc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5f0813f5f0 0x7f5f080ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f107cc640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f08140520 con 0x7f5f0813f5f0 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f107cc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f080cd7f0 con 0x7f5f08071d90 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f107cc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0813d2b0 con 0x7f5f0815adb0 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0ed69640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5f0813f5f0 0x7f5f080ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0ed69640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5f0813f5f0 0x7f5f080ce200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36888/0 (socket says 172.21.15.28:36888) 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0ed69640 1 -- 172.21.15.28:0/485644271 learned_addr learned my addr 172.21.15.28:0/485644271 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0dd67640 1 --2- 172.21.15.28:0/485644271 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f0815adb0 0x7f5f0813d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0e568640 1 --2- 172.21.15.28:0/485644271 >> v2:172.21.15.122:3300/0 conn(0x7f5f08071d90 0x7f5f08072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0ed69640 1 -- 172.21.15.28:0/485644271 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f0815adb0 msgr2=0x7f5f0813d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:42.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0ed69640 1 --2- 172.21.15.28:0/485644271 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f0815adb0 0x7f5f0813d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0ed69640 1 -- 172.21.15.28:0/485644271 >> v2:172.21.15.122:3300/0 conn(0x7f5f08071d90 msgr2=0x7f5f08072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:42.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0ed69640 1 --2- 172.21.15.28:0/485644271 >> v2:172.21.15.122:3300/0 conn(0x7f5f08071d90 0x7f5f08072170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.669+0000 7f5f0ed69640 1 -- 172.21.15.28:0/485644271 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f080cdd70 con 0x7f5f0813f5f0 2023-12-16T21:58:42.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0ed69640 1 --2- 172.21.15.28:0/485644271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0813f5f0 0x7f5f080ce200 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b1d80 tx=0x7f5f0001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18eb31698e45a035 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:42.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0d566640 1 -- 172.21.15.28:0/485644271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f000a5850 con 0x7f5f0813f5f0 2023-12-16T21:58:42.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0d566640 1 -- 172.21.15.28:0/485644271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f000a59b0 con 0x7f5f0813f5f0 2023-12-16T21:58:42.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0d566640 1 -- 172.21.15.28:0/485644271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f000a5d40 con 0x7f5f0813f5f0 2023-12-16T21:58:42.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- 172.21.15.28:0/485644271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0813f5f0 msgr2=0x7f5f080ce200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 --2- 172.21.15.28:0/485644271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0813f5f0 0x7f5f080ce200 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5f080b1d80 tx=0x7f5f0001c9d0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- 172.21.15.28:0/485644271 shutdown_connections 2023-12-16T21:58:42.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 --2- 172.21.15.28:0/485644271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0813f5f0 0x7f5f080ce200 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 --2- 172.21.15.28:0/485644271 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f0815adb0 0x7f5f0813d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 --2- 172.21.15.28:0/485644271 >> v2:172.21.15.122:3300/0 conn(0x7f5f08071d90 0x7f5f08072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- 172.21.15.28:0/485644271 >> 172.21.15.28:0/485644271 conn(0x7f5f080b0cc0 msgr2=0x7f5f080b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:42.677 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- 172.21.15.28:0/485644271 shutdown_connections 2023-12-16T21:58:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- 172.21.15.28:0/485644271 wait complete. 2023-12-16T21:58:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 Processor -- start 2023-12-16T21:58:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- start start 2023-12-16T21:58:42.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f08071d90 0x7f5f08143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f0813f5f0 0x7f5f0813c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0815adb0 0x7f5f08166230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08160dd0 con 0x7f5f0815adb0 2023-12-16T21:58:42.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080b4300 con 0x7f5f0813f5f0 2023-12-16T21:58:42.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f107cc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080b45e0 con 0x7f5f08071d90 2023-12-16T21:58:42.680 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0ed69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0815adb0 0x7f5f08166230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0ed69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0815adb0 0x7f5f08166230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36902/0 (socket says 172.21.15.28:36902) 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0ed69640 1 -- 172.21.15.28:0/2277791092 learned_addr learned my addr 172.21.15.28:0/2277791092 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0dd67640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f0813f5f0 0x7f5f0813c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0e568640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f08071d90 0x7f5f08143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0ed69640 1 -- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f08071d90 msgr2=0x7f5f08143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0ed69640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f08071d90 0x7f5f08143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0ed69640 1 -- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f0813f5f0 msgr2=0x7f5f0813c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.673+0000 7f5f0ed69640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f0813f5f0 0x7f5f0813c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5f0ed69640 1 -- 172.21.15.28:0/2277791092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f00009b30 con 0x7f5f0815adb0 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5f0ed69640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0815adb0 0x7f5f08166230 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5f000af040 tx=0x7f5f00019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5efeffd640 1 -- 172.21.15.28:0/2277791092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f000b8880 con 0x7f5f0815adb0 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5efeffd640 1 -- 172.21.15.28:0/2277791092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f000b8a00 con 0x7f5f0815adb0 2023-12-16T21:58:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f080b48c0 con 0x7f5f0815adb0 2023-12-16T21:58:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5efeffd640 1 -- 172.21.15.28:0/2277791092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f000b8db0 con 0x7f5f0815adb0 2023-12-16T21:58:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f08165cc0 con 0x7f5f0815adb0 2023-12-16T21:58:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0815d3a0 con 0x7f5f0815adb0 2023-12-16T21:58:42.682 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5efeffd640 1 -- 172.21.15.28:0/2277791092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f5f00007470 con 0x7f5f0815adb0 2023-12-16T21:58:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5efeffd640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5eec0664a0 0x7f5eec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5efeffd640 1 -- 172.21.15.28:0/2277791092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f0011f7b0 con 0x7f5f0815adb0 2023-12-16T21:58:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5f0e568640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5eec0664a0 0x7f5eec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:42.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.677+0000 7f5f0e568640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5eec0664a0 0x7f5eec068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400b9d0 tx=0x7f5f04008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:42.686 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.681+0000 7f5efeffd640 1 -- 172.21.15.28:0/2277791092 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f5f000ecc40 con 0x7f5f0815adb0 2023-12-16T21:58:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:42.865+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5f080693d0 con 0x7f5f0815adb0 2023-12-16T21:58:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5efeffd640 1 -- 172.21.15.28:0/2277791092 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5f000f0260 con 0x7f5f0815adb0 2023-12-16T21:58:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5eec0664a0 msgr2=0x7f5eec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5eec0664a0 0x7f5eec068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400b9d0 tx=0x7f5f04008040 comp rx=0 tx=0).stop 2023-12-16T21:58:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0815adb0 msgr2=0x7f5f08166230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0815adb0 0x7f5f08166230 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5f000af040 tx=0x7f5f00019770 comp rx=0 tx=0).stop 2023-12-16T21:58:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 shutdown_connections 2023-12-16T21:58:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5eec0664a0 0x7f5eec068960 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0815adb0 0x7f5f08166230 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5f0813f5f0 0x7f5f0813c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 --2- 172.21.15.28:0/2277791092 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5f08071d90 0x7f5f08143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 >> 172.21.15.28:0/2277791092 conn(0x7f5f080b0cc0 msgr2=0x7f5f0813ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 shutdown_connections 2023-12-16T21:58:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.533+0000 7f5f107cc640 1 -- 172.21.15.28:0/2277791092 wait complete. 2023-12-16T21:58:43.562 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-16T21:58:43.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:58:43.687 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.681+0000 7f71f08b4640 1 Processor -- start 2023-12-16T21:58:43.688 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.681+0000 7f71f08b4640 1 -- start start 2023-12-16T21:58:43.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71f08b4640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f71ec06c9d0 0x7f71ec070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71f08b4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec072180 0x7f71ec0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71f08b4640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ec0710e0 con 0x7f71ec144840 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71f08b4640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ec15f010 con 0x7f71ec06c9d0 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71f08b4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec070ee0 con 0x7f71ec072180 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ea59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f71ec144840 0x7f71ec06c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57370/0 (socket says 172.21.15.28:57370) 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ea59c640 1 -- 172.21.15.28:0/2058549940 learned_addr learned my addr 172.21.15.28:0/2058549940 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ead9d640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec072180 0x7f71ec0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204351404 0 0) 0x7f71ec0710e0 con 0x7f71ec144840 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e9d9b640 1 --2- 172.21.15.28:0/2058549940 >> v2:172.21.15.122:3300/0 conn(0x7f71ec06c9d0 0x7f71ec070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d8003530 con 0x7f71ec144840 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2093387036 0 0) 0x7f71d8003530 con 0x7f71ec144840 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec072180 msgr2=0x7f71ec0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:43.691 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec072180 0x7f71ec0709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 >> v2:172.21.15.122:3300/0 conn(0x7f71ec06c9d0 msgr2=0x7f71ec070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 --2- 172.21.15.28:0/2058549940 >> v2:172.21.15.122:3300/0 conn(0x7f71ec06c9d0 0x7f71ec070460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec0713c0 con 0x7f71ec144840 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f71dc001a00 con 0x7f71ec144840 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 >> v1:172.21.15.28:6789/0 conn(0x7f71ec144840 legacy=0x7f71ec06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d8003980 0x7f71d8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec072180 0x7f71d80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71d8004d10 0x7f71d8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec070ee0 con 0x7f71d8003980 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d800b990 con 0x7f71d8004d10 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d800bad0 con 0x7f71ec072180 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ea59c640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d8003980 0x7f71d8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ead9d640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71d8004d10 0x7f71d8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71e9d9b640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec072180 0x7f71d80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ea59c640 1 -- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec072180 msgr2=0x7f71d80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ea59c640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec072180 0x7f71d80047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ea59c640 1 -- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71d8004d10 msgr2=0x7f71d8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ea59c640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71d8004d10 0x7f71d8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ea59c640 1 -- 172.21.15.28:0/2058549940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d8003530 con 0x7f71d8003980 2023-12-16T21:58:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.685+0000 7f71ead9d640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71d8004d10 0x7f71d8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:43.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71ea59c640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d8003980 0x7f71d8003da0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00a8d0 tx=0x7f71dc01d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d44425163afdd7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:43.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc0ad630 con 0x7f71d8003980 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc0a8070 con 0x7f71d8003980 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e959a640 1 -- 172.21.15.28:0/2058549940 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc0ad9d0 con 0x7f71d8003980 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- 172.21.15.28:0/2058549940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d8003980 msgr2=0x7f71d8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 --2- 172.21.15.28:0/2058549940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71d8003980 0x7f71d8003da0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00a8d0 tx=0x7f71dc01d3b0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71ead9d640 1 -- 172.21.15.28:0/2058549940 reap_dead start 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- 172.21.15.28:0/2058549940 shutdown_connections 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- 172.21.15.28:0/2058549940 >> 172.21.15.28:0/2058549940 conn(0x7f71ec06a530 msgr2=0x7f71ec06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- 172.21.15.28:0/2058549940 shutdown_connections 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- 172.21.15.28:0/2058549940 wait complete. 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 Processor -- start 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- start start 2023-12-16T21:58:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71ec13ce10 0x7f71ec1626d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ec13d490 0x7f71ec0c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec13f6f0 0x7f71ec13fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec1400b0 con 0x7f71ec13d490 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec162ef0 con 0x7f71ec13ce10 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71f08b4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ec0c3fd0 con 0x7f71ec13f6f0 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ec13d490 0x7f71ec0c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ec13d490 0x7f71ec0c4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36916/0 (socket says 172.21.15.28:36916) 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 -- 172.21.15.28:0/765809373 learned_addr learned my addr 172.21.15.28:0/765809373 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71ead9d640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec13f6f0 0x7f71ec13fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 -- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec13f6f0 msgr2=0x7f71ec13fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71ea59c640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71ec13ce10 0x7f71ec1626d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec13f6f0 0x7f71ec13fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 -- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71ec13ce10 msgr2=0x7f71ec1626d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:43.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71ec13ce10 0x7f71ec1626d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 -- 172.21.15.28:0/765809373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec0c42b0 con 0x7f71ec13d490 2023-12-16T21:58:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.689+0000 7f71e9d9b640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ec13d490 0x7f71ec0c4ab0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f71d4010900 tx=0x7f71d4010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71e2ffd640 1 -- 172.21.15.28:0/765809373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4009650 con 0x7f71ec13d490 2023-12-16T21:58:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71e2ffd640 1 -- 172.21.15.28:0/765809373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d400ebc0 con 0x7f71ec13d490 2023-12-16T21:58:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ec0c4590 con 0x7f71ec13d490 2023-12-16T21:58:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71e2ffd640 1 -- 172.21.15.28:0/765809373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d401c400 con 0x7f71ec13d490 2023-12-16T21:58:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ec13d1f0 con 0x7f71ec13d490 2023-12-16T21:58:43.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ec069440 con 0x7f71ec13d490 2023-12-16T21:58:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71e2ffd640 1 -- 172.21.15.28:0/765809373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f71d401c560 con 0x7f71ec13d490 2023-12-16T21:58:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71e2ffd640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f71cc0664a0 0x7f71cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71e2ffd640 1 -- 172.21.15.28:0/765809373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71d40843e0 con 0x7f71ec13d490 2023-12-16T21:58:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71ea59c640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f71cc0664a0 0x7f71cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:43.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.693+0000 7f71ea59c640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f71cc0664a0 0x7f71cc068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0ad000 tx=0x7f71dc007040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:43.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.697+0000 7f71e2ffd640 1 -- 172.21.15.28:0/765809373 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f71d4051860 con 0x7f71ec13d490 2023-12-16T21:58:43.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:58:43.864 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:58:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.885+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71ec0c4750 con 0x7f71ec13d490 2023-12-16T21:58:43.892 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71e2ffd640 1 -- 172.21.15.28:0/765809373 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f71d4054e80 con 0x7f71ec13d490 2023-12-16T21:58:43.892 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:43.892 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/992095456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":992095456},{"type":"v1","addr":"172.21.15.122:6835","nonce":992095456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3174619236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3174619236},{"type":"v1","addr":"172.21.15.122:6837","nonce":3174619236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4691,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3342817962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3342817962},{"type":"v1","addr":"172.21.15.28:6835","nonce":3342817962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:58:40.841493+0000","modified":"2023-12-16T21:58:42.871930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5193},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5193":{"gid":5193,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/924398795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":924398795},{"type":"v1","addr":"172.21.15.28:6833","nonce":924398795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:58:43.892 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 15 2023-12-16T21:58:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f71cc0664a0 msgr2=0x7f71cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f71cc0664a0 0x7f71cc068960 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0ad000 tx=0x7f71dc007040 comp rx=0 tx=0).stop 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ec13d490 msgr2=0x7f71ec0c4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ec13d490 0x7f71ec0c4ab0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f71d4010900 tx=0x7f71d4010dc0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 shutdown_connections 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f71cc0664a0 0x7f71cc068960 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f71ec13f6f0 0x7f71ec13fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ec13d490 0x7f71ec0c4ab0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 --2- 172.21.15.28:0/765809373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71ec13ce10 0x7f71ec1626d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 >> 172.21.15.28:0/765809373 conn(0x7f71ec06a530 msgr2=0x7f71ec15ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 shutdown_connections 2023-12-16T21:58:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:43.889+0000 7f71f08b4640 1 -- 172.21.15.28:0/765809373 wait complete. 2023-12-16T21:58:43.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:58:44.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.065+0000 7f09ef5ae640 1 Processor -- start 2023-12-16T21:58:44.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ef5ae640 1 -- start start 2023-12-16T21:58:44.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ef5ae640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09e813cf30 0x7f09e806c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ef5ae640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f09e806c800 0x7f09e815a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ef5ae640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e815b3a0 con 0x7f09e815ab70 2023-12-16T21:58:44.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ef5ae640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e813d310 con 0x7f09e813cf30 2023-12-16T21:58:44.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ef5ae640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e81404e0 con 0x7f09e806c800 2023-12-16T21:58:44.073 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ed34a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09e813cf30 0x7f09e806c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09edb4b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f09e815ab70 0x7f09e8071570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57374/0 (socket says 172.21.15.28:57374) 2023-12-16T21:58:44.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09edb4b640 1 -- 172.21.15.28:0/683499338 learned_addr learned my addr 172.21.15.28:0/683499338 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:44.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ed34a640 1 -- 172.21.15.28:0/683499338 >> v2:172.21.15.122:3300/0 conn(0x7f09e806c800 msgr2=0x7f09e815a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:44.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ed34a640 1 --2- 172.21.15.28:0/683499338 >> v2:172.21.15.122:3300/0 conn(0x7f09e806c800 0x7f09e815a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ed34a640 1 -- 172.21.15.28:0/683499338 >> v1:172.21.15.28:6789/0 conn(0x7f09e815ab70 legacy=0x7f09e8071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09dffff640 1 -- 172.21.15.28:0/683499338 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008278016 0 0) 0x7f09e815b3a0 con 0x7f09e815ab70 2023-12-16T21:58:44.074 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ed34a640 1 -- 172.21.15.28:0/683499338 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e813f720 con 0x7f09e813cf30 2023-12-16T21:58:44.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.069+0000 7f09ed34a640 1 --2- 172.21.15.28:0/683499338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e813cf30 0x7f09e806c2c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f09e80cc790 tx=0x7f09e401e5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694bfe76d32fe601 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09dffff640 1 -- 172.21.15.28:0/683499338 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40ac620 con 0x7f09e813cf30 2023-12-16T21:58:44.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09dffff640 1 -- 172.21.15.28:0/683499338 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e40ac780 con 0x7f09e813cf30 2023-12-16T21:58:44.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09dffff640 1 -- 172.21.15.28:0/683499338 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40acb10 con 0x7f09e813cf30 2023-12-16T21:58:44.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- 172.21.15.28:0/683499338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e813cf30 msgr2=0x7f09e806c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.076 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/683499338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e813cf30 0x7f09e806c2c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f09e80cc790 tx=0x7f09e401e5c0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- 172.21.15.28:0/683499338 shutdown_connections 2023-12-16T21:58:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/683499338 >> v2:172.21.15.122:3300/0 conn(0x7f09e806c800 0x7f09e815a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/683499338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e813cf30 0x7f09e806c2c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- 172.21.15.28:0/683499338 >> 172.21.15.28:0/683499338 conn(0x7f09e806a530 msgr2=0x7f09e806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- 172.21.15.28:0/683499338 shutdown_connections 2023-12-16T21:58:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- 172.21.15.28:0/683499338 wait complete. 2023-12-16T21:58:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 Processor -- start 2023-12-16T21:58:44.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- start start 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e806c800 0x7f09e80716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09e813cf30 0x7f09e814f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e815ab70 0x7f09e814ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e814f900 con 0x7f09e806c800 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e814fb10 con 0x7f09e815ab70 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ef5ae640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e814f010 con 0x7f09e813cf30 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e806c800 0x7f09e80716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e806c800 0x7f09e80716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36934/0 (socket says 172.21.15.28:36934) 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 -- 172.21.15.28:0/3477474661 learned_addr learned my addr 172.21.15.28:0/3477474661 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 -- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09e813cf30 msgr2=0x7f09e814f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09e813cf30 0x7f09e814f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 -- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e815ab70 msgr2=0x7f09e814ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e815ab70 0x7f09e814ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 -- 172.21.15.28:0/3477474661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4009420 con 0x7f09e806c800 2023-12-16T21:58:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ed34a640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e806c800 0x7f09e80716a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f09e40ae040 tx=0x7f09e401ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.073+0000 7f09ddffb640 1 -- 172.21.15.28:0/3477474661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40c0410 con 0x7f09e806c800 2023-12-16T21:58:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ddffb640 1 -- 172.21.15.28:0/3477474661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e40c05b0 con 0x7f09e806c800 2023-12-16T21:58:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e814e640 con 0x7f09e806c800 2023-12-16T21:58:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ddffb640 1 -- 172.21.15.28:0/3477474661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e40c0960 con 0x7f09e806c800 2023-12-16T21:58:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e814dff0 con 0x7f09e806c800 2023-12-16T21:58:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e8069440 con 0x7f09e806c800 2023-12-16T21:58:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ddffb640 1 -- 172.21.15.28:0/3477474661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f09e40c0ac0 con 0x7f09e806c800 2023-12-16T21:58:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ddffb640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f09cc066470 0x7f09cc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ddffb640 1 -- 172.21.15.28:0/3477474661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09e4120a80 con 0x7f09e806c800 2023-12-16T21:58:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ecb49640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f09cc066470 0x7f09cc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.077+0000 7f09ecb49640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f09cc066470 0x7f09cc068930 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f09e8154ae0 tx=0x7f09d8006c20 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.085 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.081+0000 7f09ddffb640 1 -- 172.21.15.28:0/3477474661 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f09e40edf00 con 0x7f09e806c800 2023-12-16T21:58:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.237+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09e806d1e0 con 0x7f09e806c800 2023-12-16T21:58:44.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.241+0000 7f09ddffb640 1 -- 172.21.15.28:0/3477474661 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f09e40f1520 con 0x7f09e806c800 2023-12-16T21:58:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:44.246 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:42.498586+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:58:38.932657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:58:39.841920+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:58:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f09cc066470 msgr2=0x7f09cc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f09cc066470 0x7f09cc068930 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f09e8154ae0 tx=0x7f09d8006c20 comp rx=0 tx=0).stop 2023-12-16T21:58:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e806c800 msgr2=0x7f09e80716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e806c800 0x7f09e80716a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f09e40ae040 tx=0x7f09e401ce00 comp rx=0 tx=0).stop 2023-12-16T21:58:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 shutdown_connections 2023-12-16T21:58:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f09cc066470 0x7f09cc068930 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09e815ab70 0x7f09e814ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09e813cf30 0x7f09e814f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 --2- 172.21.15.28:0/3477474661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e806c800 0x7f09e80716a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 >> 172.21.15.28:0/3477474661 conn(0x7f09e806a530 msgr2=0x7f09e813e500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 shutdown_connections 2023-12-16T21:58:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.245+0000 7f09ef5ae640 1 -- 172.21.15.28:0/3477474661 wait complete. 2023-12-16T21:58:44.265 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-16T21:58:44.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:58:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8b08f5640 1 Processor -- start 2023-12-16T21:58:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8b08f5640 1 -- start start 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8b08f5640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 0x7fb8ac165fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8b08f5640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0710e0 con 0x7fb8ac06c9d0 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8b08f5640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0713c0 con 0x7fb8ac144840 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8b08f5640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac15f010 con 0x7fb8ac070b70 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a9d9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb8ac06c9d0 0x7fb8ac070460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57388/0 (socket says 172.21.15.28:57388) 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a9d9b640 1 -- 172.21.15.28:0/744586777 learned_addr learned my addr 172.21.15.28:0/744586777 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aad9d640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 0x7fb8ac165fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636741259 0 0) 0x7fb8ac0710e0 con 0x7fb8ac06c9d0 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894003530 con 0x7fb8ac06c9d0 2023-12-16T21:58:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2915617788 0 0) 0x7fb894003530 con 0x7fb8ac06c9d0 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 msgr2=0x7fb8ac165fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 0x7fb8ac165fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 >> v1:172.21.15.122:6789/0 conn(0x7fb8ac144840 legacy=0x7fb8ac06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0716a0 con 0x7fb8ac06c9d0 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fb8a0001680 con 0x7fb8ac06c9d0 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 >> v1:172.21.15.28:6789/0 conn(0x7fb8ac06c9d0 legacy=0x7fb8ac070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb894003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8940047d0 0x7fb894004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 0x7fb894005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89400ba40 con 0x7fb8940039b0 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89400bb80 con 0x7fb8940047d0 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89400bd40 con 0x7fb8ac070b70 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aa59c640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb894003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aad9d640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 0x7fb894005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8a9d9b640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8940047d0 0x7fb894004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aa59c640 1 -- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 msgr2=0x7fb894005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aa59c640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 0x7fb894005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aa59c640 1 -- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8940047d0 msgr2=0x7fb894004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aa59c640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8940047d0 0x7fb894004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aa59c640 1 -- 172.21.15.28:0/744586777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894003530 con 0x7fb8940039b0 2023-12-16T21:58:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.373+0000 7fb8aad9d640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac070b70 0x7fb894005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8aa59c640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb894003dd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb89c001750 tx=0x7fb89c01c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e639ca589f43855f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0052d0 con 0x7fb8940039b0 2023-12-16T21:58:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c0a7050 con 0x7fb8940039b0 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8a959a640 1 -- 172.21.15.28:0/744586777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c003600 con 0x7fb8940039b0 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- 172.21.15.28:0/744586777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 msgr2=0x7fb894003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 --2- 172.21.15.28:0/744586777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb894003dd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb89c001750 tx=0x7fb89c01c370 comp rx=0 tx=0).stop 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8aad9d640 1 -- 172.21.15.28:0/744586777 reap_dead start 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- 172.21.15.28:0/744586777 shutdown_connections 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- 172.21.15.28:0/744586777 >> 172.21.15.28:0/744586777 conn(0x7fb8ac06a530 msgr2=0x7fb8ac06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- 172.21.15.28:0/744586777 shutdown_connections 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- 172.21.15.28:0/744586777 wait complete. 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 Processor -- start 2023-12-16T21:58:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- start start 2023-12-16T21:58:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb8ac163a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac15a410 0x7fb8ac15c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8ac159810 0x7fb8ac159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac140cf0 con 0x7fb8940039b0 2023-12-16T21:58:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac1600a0 con 0x7fb8ac159810 2023-12-16T21:58:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8b08f5640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac15a1d0 con 0x7fb8ac15a410 2023-12-16T21:58:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8aa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb8ac163a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8aa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb8ac163a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36944/0 (socket says 172.21.15.28:36944) 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8aa59c640 1 -- 172.21.15.28:0/3149392670 learned_addr learned my addr 172.21.15.28:0/3149392670 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8aad9d640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8ac159810 0x7fb8ac159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.377+0000 7fb8a9d9b640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac15a410 0x7fb8ac15c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8aa59c640 1 -- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac15a410 msgr2=0x7fb8ac15c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8aa59c640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac15a410 0x7fb8ac15c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8aa59c640 1 -- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8ac159810 msgr2=0x7fb8ac159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8aa59c640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8ac159810 0x7fb8ac159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8aa59c640 1 -- 172.21.15.28:0/3149392670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac140730 con 0x7fb8940039b0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8aa59c640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb8ac163a90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb89c001750 tx=0x7fb894005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb89affd640 1 -- 172.21.15.28:0/3149392670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c007040 con 0x7fb8940039b0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb89affd640 1 -- 172.21.15.28:0/3149392670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c0071a0 con 0x7fb8940039b0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac140a10 con 0x7fb8940039b0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb89affd640 1 -- 172.21.15.28:0/3149392670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c007510 con 0x7fb8940039b0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac15abf0 con 0x7fb8940039b0 2023-12-16T21:58:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac069440 con 0x7fb8940039b0 2023-12-16T21:58:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb89affd640 1 -- 172.21.15.28:0/3149392670 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fb89c0067e0 con 0x7fb8940039b0 2023-12-16T21:58:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb89affd640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb88c0664a0 0x7fb88c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb89affd640 1 -- 172.21.15.28:0/3149392670 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb89c11df90 con 0x7fb8940039b0 2023-12-16T21:58:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8a9d9b640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb88c0664a0 0x7fb88c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.387 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.381+0000 7fb8a9d9b640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb88c0664a0 0x7fb88c068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00018f0 tx=0x7fb8a0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.385+0000 7fb89affd640 1 -- 172.21.15.28:0/3149392670 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fb89c0eefe0 con 0x7fb8940039b0 2023-12-16T21:58:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.537+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8ac15ff00 con 0x7fb88c0664a0 2023-12-16T21:58:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.561+0000 7fb89affd640 1 -- 172.21.15.28:0/3149392670 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278874 (secure 0 0 0) 0x7fb8a000d050 con 0x7fb88c0664a0 2023-12-16T21:58:44.567 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:44.570 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb88c0664a0 msgr2=0x7fb88c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb88c0664a0 0x7fb88c068960 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00018f0 tx=0x7fb8a0009040 comp rx=0 tx=0).stop 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 msgr2=0x7fb8ac163a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb8ac163a90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb89c001750 tx=0x7fb894005800 comp rx=0 tx=0).stop 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 shutdown_connections 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb88c0664a0 0x7fb88c068960 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8ac159810 0x7fb8ac159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb8ac15a410 0x7fb8ac15c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 --2- 172.21.15.28:0/3149392670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8940039b0 0x7fb8ac163a90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 >> 172.21.15.28:0/3149392670 conn(0x7fb8ac06a530 msgr2=0x7fb8ac141cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 shutdown_connections 2023-12-16T21:58:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.565+0000 7fb8b08f5640 1 -- 172.21.15.28:0/3149392670 wait complete. 2023-12-16T21:58:44.588 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2023-12-16T21:58:43.402199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":132,"acting":132,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":240992,"kb_used_data":4000,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733728,"statfs":{"total":773094113280,"available":772847337472,"internally_reserved":0,"allocated":4096000,"data_stored":2284674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001090"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483829+0000","last_change":"2023-12-16T21:58:40.483829+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483829+0000","last_undegraded":"2023-12-16T21:58:40.483829+0000","last_fullsized":"2023-12-16T21:58:40.483829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.492190+0000","last_change":"2023-12-16T21:58:40.492190+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.492190+0000","last_undegraded":"2023-12-16T21:58:40.492190+0000","last_fullsized":"2023-12-16T21:58:40.492190+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.499126+0000","last_change":"2023-12-16T21:58:40.499126+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.499126+0000","last_undegraded":"2023-12-16T21:58:40.499126+0000","last_fullsized":"2023-12-16T21:58:40.499126+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.505046+0000","last_change":"2023-12-16T21:58:40.505046+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.505046+0000","last_undegraded":"2023-12-16T21:58:40.505046+0000","last_fullsized":"2023-12-16T21:58:40.505046+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.488032+0000","last_change":"2023-12-16T21:58:40.488032+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.488032+0000","last_undegraded":"2023-12-16T21:58:40.488032+0000","last_fullsized":"2023-12-16T21:58:40.488032+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494042+0000","last_change":"2023-12-16T21:58:40.494042+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494042+0000","last_undegraded":"2023-12-16T21:58:40.494042+0000","last_fullsized":"2023-12-16T21:58:40.494042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.492852+0000","last_change":"2023-12-16T21:58:40.492852+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.492852+0000","last_undegraded":"2023-12-16T21:58:40.492852+0000","last_fullsized":"2023-12-16T21:58:40.492852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.496638+0000","last_change":"2023-12-16T21:58:40.496638+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.496638+0000","last_undegraded":"2023-12-16T21:58:40.496638+0000","last_fullsized":"2023-12-16T21:58:40.496638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493692+0000","last_change":"2023-12-16T21:58:40.493692+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493692+0000","last_undegraded":"2023-12-16T21:58:40.493692+0000","last_fullsized":"2023-12-16T21:58:40.493692+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.497617+0000","last_change":"2023-12-16T21:58:40.497617+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.497617+0000","last_undegraded":"2023-12-16T21:58:40.497617+0000","last_fullsized":"2023-12-16T21:58:40.497617+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491447+0000","last_change":"2023-12-16T21:58:40.491447+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491447+0000","last_undegraded":"2023-12-16T21:58:40.491447+0000","last_fullsized":"2023-12-16T21:58:40.491447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.487840+0000","last_change":"2023-12-16T21:58:40.487840+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.487840+0000","last_undegraded":"2023-12-16T21:58:40.487840+0000","last_fullsized":"2023-12-16T21:58:40.487840+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.490713+0000","last_change":"2023-12-16T21:58:40.490713+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.490713+0000","last_undegraded":"2023-12-16T21:58:40.490713+0000","last_fullsized":"2023-12-16T21:58:40.490713+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491975+0000","last_change":"2023-12-16T21:58:40.491975+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491975+0000","last_undegraded":"2023-12-16T21:58:40.491975+0000","last_fullsized":"2023-12-16T21:58:40.491975+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.488011+0000","last_change":"2023-12-16T21:58:40.488011+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.488011+0000","last_undegraded":"2023-12-16T21:58:40.488011+0000","last_fullsized":"2023-12-16T21:58:40.488011+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.495037+0000","last_change":"2023-12-16T21:58:40.495037+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.495037+0000","last_undegraded":"2023-12-16T21:58:40.495037+0000","last_fullsized":"2023-12-16T21:58:40.495037+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.484902+0000","last_change":"2023-12-16T21:58:40.484902+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.484902+0000","last_undegraded":"2023-12-16T21:58:40.484902+0000","last_fullsized":"2023-12-16T21:58:40.484902+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.501035+0000","last_change":"2023-12-16T21:58:40.501035+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.501035+0000","last_undegraded":"2023-12-16T21:58:40.501035+0000","last_fullsized":"2023-12-16T21:58:40.501035+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.517957+0000","last_change":"2023-12-16T21:58:40.510463+0000","last_active":"2023-12-16T21:58:40.517957+0000","last_peered":"2023-12-16T21:58:40.517957+0000","last_clean":"2023-12-16T21:58:40.517957+0000","last_became_active":"2023-12-16T21:58:40.509883+0000","last_became_peered":"2023-12-16T21:58:40.509883+0000","last_unstale":"2023-12-16T21:58:40.517957+0000","last_undegraded":"2023-12-16T21:58:40.517957+0000","last_fullsized":"2023-12-16T21:58:40.517957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:46:52.112074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.482640+0000","last_change":"2023-12-16T21:58:40.482640+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.482640+0000","last_undegraded":"2023-12-16T21:58:40.482640+0000","last_fullsized":"2023-12-16T21:58:40.482640+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.529438+0000","last_change":"2023-12-16T21:58:39.529438+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.529438+0000","last_undegraded":"2023-12-16T21:58:39.529438+0000","last_fullsized":"2023-12-16T21:58:39.529438+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.489570+0000","last_change":"2023-12-16T21:58:40.489570+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.489570+0000","last_undegraded":"2023-12-16T21:58:40.489570+0000","last_fullsized":"2023-12-16T21:58:40.489570+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478861+0000","last_change":"2023-12-16T21:58:39.478861+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478861+0000","last_undegraded":"2023-12-16T21:58:39.478861+0000","last_fullsized":"2023-12-16T21:58:39.478861+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:39.467225+0000","last_change":"2023-12-16T21:58:39.467225+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.467225+0000","last_undegraded":"2023-12-16T21:58:39.467225+0000","last_fullsized":"2023-12-16T21:58:39.467225+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493858+0000","last_change":"2023-12-16T21:58:40.493858+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493858+0000","last_undegraded":"2023-12-16T21:58:40.493858+0000","last_fullsized":"2023-12-16T21:58:40.493858+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522509+0000","last_change":"2023-12-16T21:58:40.516801+0000","last_active":"2023-12-16T21:58:40.522509+0000","last_peered":"2023-12-16T21:58:40.522509+0000","last_clean":"2023-12-16T21:58:40.522509+0000","last_became_active":"2023-12-16T21:58:40.515829+0000","last_became_peered":"2023-12-16T21:58:40.515829+0000","last_unstale":"2023-12-16T21:58:40.522509+0000","last_undegraded":"2023-12-16T21:58:40.522509+0000","last_fullsized":"2023-12-16T21:58:40.522509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:26:05.935472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494574+0000","last_change":"2023-12-16T21:58:40.494574+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494574+0000","last_undegraded":"2023-12-16T21:58:40.494574+0000","last_fullsized":"2023-12-16T21:58:40.494574+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514237+0000","last_change":"2023-12-16T21:58:40.501591+0000","last_active":"2023-12-16T21:58:40.514237+0000","last_peered":"2023-12-16T21:58:40.514237+0000","last_clean":"2023-12-16T21:58:40.514237+0000","last_became_active":"2023-12-16T21:58:40.500870+0000","last_became_peered":"2023-12-16T21:58:40.500870+0000","last_unstale":"2023-12-16T21:58:40.514237+0000","last_undegraded":"2023-12-16T21:58:40.514237+0000","last_fullsized":"2023-12-16T21:58:40.514237+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:34:30.857548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483333+0000","last_change":"2023-12-16T21:58:40.483333+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483333+0000","last_undegraded":"2023-12-16T21:58:40.483333+0000","last_fullsized":"2023-12-16T21:58:40.483333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478762+0000","last_change":"2023-12-16T21:58:39.478762+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478762+0000","last_undegraded":"2023-12-16T21:58:39.478762+0000","last_fullsized":"2023-12-16T21:58:39.478762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519509+0000","last_change":"2023-12-16T21:58:40.511738+0000","last_active":"2023-12-16T21:58:40.519509+0000","last_peered":"2023-12-16T21:58:40.519509+0000","last_clean":"2023-12-16T21:58:40.519509+0000","last_became_active":"2023-12-16T21:58:40.510752+0000","last_became_peered":"2023-12-16T21:58:40.510752+0000","last_unstale":"2023-12-16T21:58:40.519509+0000","last_undegraded":"2023-12-16T21:58:40.519509+0000","last_fullsized":"2023-12-16T21:58:40.519509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:47:01.519007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491282+0000","last_change":"2023-12-16T21:58:40.491282+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491282+0000","last_undegraded":"2023-12-16T21:58:40.491282+0000","last_fullsized":"2023-12-16T21:58:40.491282+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477459+0000","last_change":"2023-12-16T21:58:39.477459+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477459+0000","last_undegraded":"2023-12-16T21:58:39.477459+0000","last_fullsized":"2023-12-16T21:58:39.477459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528902+0000","last_change":"2023-12-16T21:58:39.528902+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528902+0000","last_undegraded":"2023-12-16T21:58:39.528902+0000","last_fullsized":"2023-12-16T21:58:39.528902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.523268+0000","last_change":"2023-12-16T21:58:40.518810+0000","last_active":"2023-12-16T21:58:40.523268+0000","last_peered":"2023-12-16T21:58:40.523268+0000","last_clean":"2023-12-16T21:58:40.523268+0000","last_became_active":"2023-12-16T21:58:40.518226+0000","last_became_peered":"2023-12-16T21:58:40.518226+0000","last_unstale":"2023-12-16T21:58:40.523268+0000","last_undegraded":"2023-12-16T21:58:40.523268+0000","last_fullsized":"2023-12-16T21:58:40.523268+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:37:11.035240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493053+0000","last_change":"2023-12-16T21:58:40.493053+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493053+0000","last_undegraded":"2023-12-16T21:58:40.493053+0000","last_fullsized":"2023-12-16T21:58:40.493053+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477394+0000","last_change":"2023-12-16T21:58:39.477394+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477394+0000","last_undegraded":"2023-12-16T21:58:39.477394+0000","last_fullsized":"2023-12-16T21:58:39.477394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519501+0000","last_change":"2023-12-16T21:58:40.511320+0000","last_active":"2023-12-16T21:58:40.519501+0000","last_peered":"2023-12-16T21:58:40.519501+0000","last_clean":"2023-12-16T21:58:40.519501+0000","last_became_active":"2023-12-16T21:58:40.510584+0000","last_became_peered":"2023-12-16T21:58:40.510584+0000","last_unstale":"2023-12-16T21:58:40.519501+0000","last_undegraded":"2023-12-16T21:58:40.519501+0000","last_fullsized":"2023-12-16T21:58:40.519501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:36.075932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.482247+0000","last_change":"2023-12-16T21:58:39.482247+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.482247+0000","last_undegraded":"2023-12-16T21:58:39.482247+0000","last_fullsized":"2023-12-16T21:58:39.482247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.496004+0000","last_change":"2023-12-16T21:58:40.496004+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.496004+0000","last_undegraded":"2023-12-16T21:58:40.496004+0000","last_fullsized":"2023-12-16T21:58:40.496004+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477445+0000","last_change":"2023-12-16T21:58:39.477445+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477445+0000","last_undegraded":"2023-12-16T21:58:39.477445+0000","last_fullsized":"2023-12-16T21:58:39.477445+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.482543+0000","last_change":"2023-12-16T21:58:40.482543+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.482543+0000","last_undegraded":"2023-12-16T21:58:40.482543+0000","last_fullsized":"2023-12-16T21:58:40.482543+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478219+0000","last_change":"2023-12-16T21:58:39.478219+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478219+0000","last_undegraded":"2023-12-16T21:58:39.478219+0000","last_fullsized":"2023-12-16T21:58:39.478219+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.485868+0000","last_change":"2023-12-16T21:58:40.485868+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.485868+0000","last_undegraded":"2023-12-16T21:58:40.485868+0000","last_fullsized":"2023-12-16T21:58:40.485868+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.481593+0000","last_change":"2023-12-16T21:58:39.481593+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.481593+0000","last_undegraded":"2023-12-16T21:58:39.481593+0000","last_fullsized":"2023-12-16T21:58:39.481593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494957+0000","last_change":"2023-12-16T21:58:40.494957+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494957+0000","last_undegraded":"2023-12-16T21:58:40.494957+0000","last_fullsized":"2023-12-16T21:58:40.494957+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522371+0000","last_change":"2023-12-16T21:58:40.515564+0000","last_active":"2023-12-16T21:58:40.522371+0000","last_peered":"2023-12-16T21:58:40.522371+0000","last_clean":"2023-12-16T21:58:40.522371+0000","last_became_active":"2023-12-16T21:58:40.514839+0000","last_became_peered":"2023-12-16T21:58:40.514839+0000","last_unstale":"2023-12-16T21:58:40.522371+0000","last_undegraded":"2023-12-16T21:58:40.522371+0000","last_fullsized":"2023-12-16T21:58:40.522371+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:58:46.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514447+0000","last_change":"2023-12-16T21:58:40.499987+0000","last_active":"2023-12-16T21:58:40.514447+0000","last_peered":"2023-12-16T21:58:40.514447+0000","last_clean":"2023-12-16T21:58:40.514447+0000","last_became_active":"2023-12-16T21:58:40.499198+0000","last_became_peered":"2023-12-16T21:58:40.499198+0000","last_unstale":"2023-12-16T21:58:40.514447+0000","last_undegraded":"2023-12-16T21:58:40.514447+0000","last_fullsized":"2023-12-16T21:58:40.514447+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:53.492867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514335+0000","last_change":"2023-12-16T21:58:40.503566+0000","last_active":"2023-12-16T21:58:40.514335+0000","last_peered":"2023-12-16T21:58:40.514335+0000","last_clean":"2023-12-16T21:58:40.514335+0000","last_became_active":"2023-12-16T21:58:40.502798+0000","last_became_peered":"2023-12-16T21:58:40.502798+0000","last_unstale":"2023-12-16T21:58:40.514335+0000","last_undegraded":"2023-12-16T21:58:40.514335+0000","last_fullsized":"2023-12-16T21:58:40.514335+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:06:31.466515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528972+0000","last_change":"2023-12-16T21:58:39.528972+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528972+0000","last_undegraded":"2023-12-16T21:58:39.528972+0000","last_fullsized":"2023-12-16T21:58:39.528972+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.512812+0000","last_change":"2023-12-16T21:58:40.502158+0000","last_active":"2023-12-16T21:58:40.512812+0000","last_peered":"2023-12-16T21:58:40.512812+0000","last_clean":"2023-12-16T21:58:40.512812+0000","last_became_active":"2023-12-16T21:58:40.501224+0000","last_became_peered":"2023-12-16T21:58:40.501224+0000","last_unstale":"2023-12-16T21:58:40.512812+0000","last_undegraded":"2023-12-16T21:58:40.512812+0000","last_fullsized":"2023-12-16T21:58:40.512812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:11:13.641738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514394+0000","last_change":"2023-12-16T21:58:40.503347+0000","last_active":"2023-12-16T21:58:40.514394+0000","last_peered":"2023-12-16T21:58:40.514394+0000","last_clean":"2023-12-16T21:58:40.514394+0000","last_became_active":"2023-12-16T21:58:40.502397+0000","last_became_peered":"2023-12-16T21:58:40.502397+0000","last_unstale":"2023-12-16T21:58:40.514394+0000","last_undegraded":"2023-12-16T21:58:40.514394+0000","last_fullsized":"2023-12-16T21:58:40.514394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:12:26.461737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.500732+0000","last_change":"2023-12-16T21:58:40.500732+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.500732+0000","last_undegraded":"2023-12-16T21:58:40.500732+0000","last_fullsized":"2023-12-16T21:58:40.500732+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.523302+0000","last_change":"2023-12-16T21:58:40.519887+0000","last_active":"2023-12-16T21:58:40.523302+0000","last_peered":"2023-12-16T21:58:40.523302+0000","last_clean":"2023-12-16T21:58:40.523302+0000","last_became_active":"2023-12-16T21:58:40.518760+0000","last_became_peered":"2023-12-16T21:58:40.518760+0000","last_unstale":"2023-12-16T21:58:40.523302+0000","last_undegraded":"2023-12-16T21:58:40.523302+0000","last_fullsized":"2023-12-16T21:58:40.523302+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:18:57.185826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480909+0000","last_change":"2023-12-16T21:58:39.480909+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480909+0000","last_undegraded":"2023-12-16T21:58:39.480909+0000","last_fullsized":"2023-12-16T21:58:39.480909+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494047+0000","last_change":"2023-12-16T21:58:40.494047+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494047+0000","last_undegraded":"2023-12-16T21:58:40.494047+0000","last_fullsized":"2023-12-16T21:58:40.494047+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528116+0000","last_change":"2023-12-16T21:58:39.528116+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528116+0000","last_undegraded":"2023-12-16T21:58:39.528116+0000","last_fullsized":"2023-12-16T21:58:39.528116+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480710+0000","last_change":"2023-12-16T21:58:39.480710+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480710+0000","last_undegraded":"2023-12-16T21:58:39.480710+0000","last_fullsized":"2023-12-16T21:58:39.480710+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483887+0000","last_change":"2023-12-16T21:58:40.483887+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483887+0000","last_undegraded":"2023-12-16T21:58:40.483887+0000","last_fullsized":"2023-12-16T21:58:40.483887+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528182+0000","last_change":"2023-12-16T21:58:39.528182+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528182+0000","last_undegraded":"2023-12-16T21:58:39.528182+0000","last_fullsized":"2023-12-16T21:58:39.528182+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480584+0000","last_change":"2023-12-16T21:58:39.480584+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480584+0000","last_undegraded":"2023-12-16T21:58:39.480584+0000","last_fullsized":"2023-12-16T21:58:39.480584+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522536+0000","last_change":"2023-12-16T21:58:40.516094+0000","last_active":"2023-12-16T21:58:40.522536+0000","last_peered":"2023-12-16T21:58:40.522536+0000","last_clean":"2023-12-16T21:58:40.522536+0000","last_became_active":"2023-12-16T21:58:40.515413+0000","last_became_peered":"2023-12-16T21:58:40.515413+0000","last_unstale":"2023-12-16T21:58:40.522536+0000","last_undegraded":"2023-12-16T21:58:40.522536+0000","last_fullsized":"2023-12-16T21:58:40.522536+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:48:35.936102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.484418+0000","last_change":"2023-12-16T21:58:40.484418+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.484418+0000","last_undegraded":"2023-12-16T21:58:40.484418+0000","last_fullsized":"2023-12-16T21:58:40.484418+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477392+0000","last_change":"2023-12-16T21:58:39.477392+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477392+0000","last_undegraded":"2023-12-16T21:58:39.477392+0000","last_fullsized":"2023-12-16T21:58:39.477392+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.497157+0000","last_change":"2023-12-16T21:58:40.497157+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.497157+0000","last_undegraded":"2023-12-16T21:58:40.497157+0000","last_fullsized":"2023-12-16T21:58:40.497157+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519500+0000","last_change":"2023-12-16T21:58:40.516743+0000","last_active":"2023-12-16T21:58:40.519500+0000","last_peered":"2023-12-16T21:58:40.519500+0000","last_clean":"2023-12-16T21:58:40.519500+0000","last_became_active":"2023-12-16T21:58:40.515578+0000","last_became_peered":"2023-12-16T21:58:40.515578+0000","last_unstale":"2023-12-16T21:58:40.519500+0000","last_undegraded":"2023-12-16T21:58:40.519500+0000","last_fullsized":"2023-12-16T21:58:40.519500+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:18:32.213979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:58:39.484154+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:58:39.484154+0000","last_peered":"2023-12-16T21:58:39.484154+0000","last_clean":"2023-12-16T21:58:39.484154+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:58:39.484154+0000","last_undegraded":"2023-12-16T21:58:39.484154+0000","last_fullsized":"2023-12-16T21:58:39.484154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":68,"acting":68,"num_store_stats":4},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":732948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.1779999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.26}]},{"osd":4,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.81599999999999995}]},{"osd":5,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.0620000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.109}]}]},{"osd":6,"up_from":16,"seq":68719476753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.66800000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.71899999999999997}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.754}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":1.083}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":5.5620000000000003,"5min":5.5620000000000003,"15min":5.5620000000000003},"last":1.0549999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.93200000000000005}]}]},{"osd":5,"up_from":16,"seq":68719476754,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.56200000000000006}]},{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.72999999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.0569999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.98399999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.018}]},{"osd":6,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.254,"5min":3.254,"15min":3.254},"last":0.95199999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.512,"5min":2.512,"15min":2.512},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":1.339}]}]},{"osd":4,"up_from":16,"seq":68719476752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.79100000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.91600000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.998},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.95599999999999996}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.123}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.208},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.173}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.085}]}]},{"osd":3,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.76700000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.80700000000000005}]}]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.371},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.435},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.64300000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.65900000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.87}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.70599999999999996}]}]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":942080,"data_stored":720864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.79700000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.85599999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.21},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.046}]},{"osd":5,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":1.327},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.2569999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.081,"5min":2.081,"15min":2.081},"last":1.1719999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.3779999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542161,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31528,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":348160,"data_stored":130496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.63},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.45800000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.503}]},{"osd":3,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.56699999999999995}]},{"osd":4,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":0.70299999999999996}]},{"osd":6,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.0660000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:58:44.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:58:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fcccb1aa640 1 Processor -- start 2023-12-16T21:58:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fcccb1aa640 1 -- start start 2023-12-16T21:58:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fcccb1aa640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fccc413cf30 0x7fccc406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fcccb1aa640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fccc406c800 0x7fccc415a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fcccb1aa640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc415b3a0 con 0x7fccc415ab70 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fcccb1aa640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc413d310 con 0x7fccc413cf30 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fcccb1aa640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc41404e0 con 0x7fccc406c800 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fccc8f46640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fccc413cf30 0x7fccc406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fccc8f46640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fccc413cf30 0x7fccc406c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36946/0 (socket says 172.21.15.28:36946) 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fccc8f46640 1 -- 172.21.15.28:0/2236059930 learned_addr learned my addr 172.21.15.28:0/2236059930 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fccc8f46640 1 -- 172.21.15.28:0/2236059930 >> v2:172.21.15.122:3300/0 conn(0x7fccc406c800 msgr2=0x7fccc415a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fccc3fff640 1 --2- 172.21.15.28:0/2236059930 >> v2:172.21.15.122:3300/0 conn(0x7fccc406c800 0x7fccc415a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fccc8f46640 1 --2- 172.21.15.28:0/2236059930 >> v2:172.21.15.122:3300/0 conn(0x7fccc406c800 0x7fccc415a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.757+0000 7fccc8f46640 1 -- 172.21.15.28:0/2236059930 >> v1:172.21.15.122:6789/0 conn(0x7fccc415ab70 legacy=0x7fccc4071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fccc8f46640 1 -- 172.21.15.28:0/2236059930 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc413f720 con 0x7fccc413cf30 2023-12-16T21:58:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fccc8f46640 1 --2- 172.21.15.28:0/2236059930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc413cf30 0x7fccc406c2c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fccc40cc790 tx=0x7fccb401eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a357554eeb7727 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fccc37fe640 1 -- 172.21.15.28:0/2236059930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40a7ac0 con 0x7fccc413cf30 2023-12-16T21:58:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fccc37fe640 1 -- 172.21.15.28:0/2236059930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb40a7c20 con 0x7fccc413cf30 2023-12-16T21:58:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fccc37fe640 1 -- 172.21.15.28:0/2236059930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40af440 con 0x7fccc413cf30 2023-12-16T21:58:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fcccb1aa640 1 -- 172.21.15.28:0/2236059930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc413cf30 msgr2=0x7fccc406c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/2236059930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc413cf30 0x7fccc406c2c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fccc40cc790 tx=0x7fccb401eb00 comp rx=0 tx=0).stop 2023-12-16T21:58:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fcccb1aa640 1 -- 172.21.15.28:0/2236059930 shutdown_connections 2023-12-16T21:58:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/2236059930 >> v2:172.21.15.122:3300/0 conn(0x7fccc406c800 0x7fccc415a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/2236059930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc413cf30 0x7fccc406c2c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fcccb1aa640 1 -- 172.21.15.28:0/2236059930 >> 172.21.15.28:0/2236059930 conn(0x7fccc406a530 msgr2=0x7fccc406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fcccb1aa640 1 -- 172.21.15.28:0/2236059930 shutdown_connections 2023-12-16T21:58:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.761+0000 7fcccb1aa640 1 -- 172.21.15.28:0/2236059930 wait complete. 2023-12-16T21:58:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 Processor -- start 2023-12-16T21:58:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 -- start start 2023-12-16T21:58:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccc406c800 0x7fccc4155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fccc413cf30 0x7fccc4138df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc415ab70 0x7fccc4139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40b3190 con 0x7fccc415ab70 2023-12-16T21:58:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc4071a90 con 0x7fccc406c800 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40b2e60 con 0x7fccc413cf30 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc415ab70 0x7fccc4139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc415ab70 0x7fccc4139620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36948/0 (socket says 172.21.15.28:36948) 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 -- 172.21.15.28:0/3314415316 learned_addr learned my addr 172.21.15.28:0/3314415316 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc8f46640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccc406c800 0x7fccc4155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc3fff640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fccc413cf30 0x7fccc4138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 -- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fccc413cf30 msgr2=0x7fccc4138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fccc413cf30 0x7fccc4138df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 -- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccc406c800 msgr2=0x7fccc4155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccc406c800 0x7fccc4155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 -- 172.21.15.28:0/3314415316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb400b770 con 0x7fccc415ab70 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc3fff640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fccc413cf30 0x7fccc4138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc9747640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc415ab70 0x7fccc4139620 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00b410 tx=0x7fccbc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc17fa640 1 -- 172.21.15.28:0/3314415316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc011830 con 0x7fccc415ab70 2023-12-16T21:58:44.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc17fa640 1 -- 172.21.15.28:0/3314415316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc0119d0 con 0x7fccc415ab70 2023-12-16T21:58:44.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc17fa640 1 -- 172.21.15.28:0/3314415316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc011d80 con 0x7fccc415ab70 2023-12-16T21:58:44.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc40714d0 con 0x7fccc415ab70 2023-12-16T21:58:44.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc4139330 con 0x7fccc415ab70 2023-12-16T21:58:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc40b3340 con 0x7fccc415ab70 2023-12-16T21:58:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.765+0000 7fccc17fa640 1 -- 172.21.15.28:0/3314415316 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fccbc01a400 con 0x7fccc415ab70 2023-12-16T21:58:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.769+0000 7fccc17fa640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fccb0066470 0x7fccb0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.769+0000 7fccc17fa640 1 -- 172.21.15.28:0/3314415316 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccbc083fe0 con 0x7fccc415ab70 2023-12-16T21:58:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.769+0000 7fccc8f46640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fccb0066470 0x7fccb0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.769+0000 7fccc8f46640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fccb0066470 0x7fccb0068930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fccb40b1000 tx=0x7fccb40a9070 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:44.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.773+0000 7fccc17fa640 1 -- 172.21.15.28:0/3314415316 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fccc40b3340 con 0x7fccc415ab70 2023-12-16T21:58:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.925+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fccc415afa0 con 0x7fccb0066470 2023-12-16T21:58:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fccc17fa640 1 -- 172.21.15.28:0/3314415316 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278874 (secure 0 0 0) 0x7fccb40b93e0 con 0x7fccb0066470 2023-12-16T21:58:44.953 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:58:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fccb0066470 msgr2=0x7fccb0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fccb0066470 0x7fccb0068930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fccb40b1000 tx=0x7fccb40a9070 comp rx=0 tx=0).stop 2023-12-16T21:58:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc415ab70 msgr2=0x7fccc4139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc415ab70 0x7fccc4139620 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00b410 tx=0x7fccbc00b8d0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 shutdown_connections 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fccb0066470 0x7fccb0068930 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc415ab70 0x7fccc4139620 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fccc413cf30 0x7fccc4138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 --2- 172.21.15.28:0/3314415316 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fccc406c800 0x7fccc4155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 >> 172.21.15.28:0/3314415316 conn(0x7fccc406a530 msgr2=0x7fccc413d860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 shutdown_connections 2023-12-16T21:58:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:44.949+0000 7fcccb1aa640 1 -- 172.21.15.28:0/3314415316 wait complete. 2023-12-16T21:58:44.972 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2023-12-16T21:58:43.402199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":132,"acting":132,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":240992,"kb_used_data":4000,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733728,"statfs":{"total":773094113280,"available":772847337472,"internally_reserved":0,"allocated":4096000,"data_stored":2284674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001090"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483829+0000","last_change":"2023-12-16T21:58:40.483829+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483829+0000","last_undegraded":"2023-12-16T21:58:40.483829+0000","last_fullsized":"2023-12-16T21:58:40.483829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.492190+0000","last_change":"2023-12-16T21:58:40.492190+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.492190+0000","last_undegraded":"2023-12-16T21:58:40.492190+0000","last_fullsized":"2023-12-16T21:58:40.492190+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.499126+0000","last_change":"2023-12-16T21:58:40.499126+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.499126+0000","last_undegraded":"2023-12-16T21:58:40.499126+0000","last_fullsized":"2023-12-16T21:58:40.499126+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.505046+0000","last_change":"2023-12-16T21:58:40.505046+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.505046+0000","last_undegraded":"2023-12-16T21:58:40.505046+0000","last_fullsized":"2023-12-16T21:58:40.505046+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.488032+0000","last_change":"2023-12-16T21:58:40.488032+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.488032+0000","last_undegraded":"2023-12-16T21:58:40.488032+0000","last_fullsized":"2023-12-16T21:58:40.488032+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494042+0000","last_change":"2023-12-16T21:58:40.494042+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494042+0000","last_undegraded":"2023-12-16T21:58:40.494042+0000","last_fullsized":"2023-12-16T21:58:40.494042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.492852+0000","last_change":"2023-12-16T21:58:40.492852+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.492852+0000","last_undegraded":"2023-12-16T21:58:40.492852+0000","last_fullsized":"2023-12-16T21:58:40.492852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.496638+0000","last_change":"2023-12-16T21:58:40.496638+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.496638+0000","last_undegraded":"2023-12-16T21:58:40.496638+0000","last_fullsized":"2023-12-16T21:58:40.496638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493692+0000","last_change":"2023-12-16T21:58:40.493692+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493692+0000","last_undegraded":"2023-12-16T21:58:40.493692+0000","last_fullsized":"2023-12-16T21:58:40.493692+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.497617+0000","last_change":"2023-12-16T21:58:40.497617+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.497617+0000","last_undegraded":"2023-12-16T21:58:40.497617+0000","last_fullsized":"2023-12-16T21:58:40.497617+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491447+0000","last_change":"2023-12-16T21:58:40.491447+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491447+0000","last_undegraded":"2023-12-16T21:58:40.491447+0000","last_fullsized":"2023-12-16T21:58:40.491447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.487840+0000","last_change":"2023-12-16T21:58:40.487840+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.487840+0000","last_undegraded":"2023-12-16T21:58:40.487840+0000","last_fullsized":"2023-12-16T21:58:40.487840+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.490713+0000","last_change":"2023-12-16T21:58:40.490713+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.490713+0000","last_undegraded":"2023-12-16T21:58:40.490713+0000","last_fullsized":"2023-12-16T21:58:40.490713+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491975+0000","last_change":"2023-12-16T21:58:40.491975+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491975+0000","last_undegraded":"2023-12-16T21:58:40.491975+0000","last_fullsized":"2023-12-16T21:58:40.491975+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.488011+0000","last_change":"2023-12-16T21:58:40.488011+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.488011+0000","last_undegraded":"2023-12-16T21:58:40.488011+0000","last_fullsized":"2023-12-16T21:58:40.488011+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.495037+0000","last_change":"2023-12-16T21:58:40.495037+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.495037+0000","last_undegraded":"2023-12-16T21:58:40.495037+0000","last_fullsized":"2023-12-16T21:58:40.495037+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.484902+0000","last_change":"2023-12-16T21:58:40.484902+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.484902+0000","last_undegraded":"2023-12-16T21:58:40.484902+0000","last_fullsized":"2023-12-16T21:58:40.484902+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.501035+0000","last_change":"2023-12-16T21:58:40.501035+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.501035+0000","last_undegraded":"2023-12-16T21:58:40.501035+0000","last_fullsized":"2023-12-16T21:58:40.501035+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.517957+0000","last_change":"2023-12-16T21:58:40.510463+0000","last_active":"2023-12-16T21:58:40.517957+0000","last_peered":"2023-12-16T21:58:40.517957+0000","last_clean":"2023-12-16T21:58:40.517957+0000","last_became_active":"2023-12-16T21:58:40.509883+0000","last_became_peered":"2023-12-16T21:58:40.509883+0000","last_unstale":"2023-12-16T21:58:40.517957+0000","last_undegraded":"2023-12-16T21:58:40.517957+0000","last_fullsized":"2023-12-16T21:58:40.517957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:46:52.112074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.482640+0000","last_change":"2023-12-16T21:58:40.482640+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.482640+0000","last_undegraded":"2023-12-16T21:58:40.482640+0000","last_fullsized":"2023-12-16T21:58:40.482640+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.529438+0000","last_change":"2023-12-16T21:58:39.529438+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.529438+0000","last_undegraded":"2023-12-16T21:58:39.529438+0000","last_fullsized":"2023-12-16T21:58:39.529438+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.489570+0000","last_change":"2023-12-16T21:58:40.489570+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.489570+0000","last_undegraded":"2023-12-16T21:58:40.489570+0000","last_fullsized":"2023-12-16T21:58:40.489570+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478861+0000","last_change":"2023-12-16T21:58:39.478861+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478861+0000","last_undegraded":"2023-12-16T21:58:39.478861+0000","last_fullsized":"2023-12-16T21:58:39.478861+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:39.467225+0000","last_change":"2023-12-16T21:58:39.467225+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.467225+0000","last_undegraded":"2023-12-16T21:58:39.467225+0000","last_fullsized":"2023-12-16T21:58:39.467225+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493858+0000","last_change":"2023-12-16T21:58:40.493858+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493858+0000","last_undegraded":"2023-12-16T21:58:40.493858+0000","last_fullsized":"2023-12-16T21:58:40.493858+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522509+0000","last_change":"2023-12-16T21:58:40.516801+0000","last_active":"2023-12-16T21:58:40.522509+0000","last_peered":"2023-12-16T21:58:40.522509+0000","last_clean":"2023-12-16T21:58:40.522509+0000","last_became_active":"2023-12-16T21:58:40.515829+0000","last_became_peered":"2023-12-16T21:58:40.515829+0000","last_unstale":"2023-12-16T21:58:40.522509+0000","last_undegraded":"2023-12-16T21:58:40.522509+0000","last_fullsized":"2023-12-16T21:58:40.522509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:26:05.935472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494574+0000","last_change":"2023-12-16T21:58:40.494574+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494574+0000","last_undegraded":"2023-12-16T21:58:40.494574+0000","last_fullsized":"2023-12-16T21:58:40.494574+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514237+0000","last_change":"2023-12-16T21:58:40.501591+0000","last_active":"2023-12-16T21:58:40.514237+0000","last_peered":"2023-12-16T21:58:40.514237+0000","last_clean":"2023-12-16T21:58:40.514237+0000","last_became_active":"2023-12-16T21:58:40.500870+0000","last_became_peered":"2023-12-16T21:58:40.500870+0000","last_unstale":"2023-12-16T21:58:40.514237+0000","last_undegraded":"2023-12-16T21:58:40.514237+0000","last_fullsized":"2023-12-16T21:58:40.514237+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:34:30.857548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483333+0000","last_change":"2023-12-16T21:58:40.483333+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483333+0000","last_undegraded":"2023-12-16T21:58:40.483333+0000","last_fullsized":"2023-12-16T21:58:40.483333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478762+0000","last_change":"2023-12-16T21:58:39.478762+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478762+0000","last_undegraded":"2023-12-16T21:58:39.478762+0000","last_fullsized":"2023-12-16T21:58:39.478762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519509+0000","last_change":"2023-12-16T21:58:40.511738+0000","last_active":"2023-12-16T21:58:40.519509+0000","last_peered":"2023-12-16T21:58:40.519509+0000","last_clean":"2023-12-16T21:58:40.519509+0000","last_became_active":"2023-12-16T21:58:40.510752+0000","last_became_peered":"2023-12-16T21:58:40.510752+0000","last_unstale":"2023-12-16T21:58:40.519509+0000","last_undegraded":"2023-12-16T21:58:40.519509+0000","last_fullsized":"2023-12-16T21:58:40.519509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:47:01.519007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491282+0000","last_change":"2023-12-16T21:58:40.491282+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491282+0000","last_undegraded":"2023-12-16T21:58:40.491282+0000","last_fullsized":"2023-12-16T21:58:40.491282+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477459+0000","last_change":"2023-12-16T21:58:39.477459+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477459+0000","last_undegraded":"2023-12-16T21:58:39.477459+0000","last_fullsized":"2023-12-16T21:58:39.477459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528902+0000","last_change":"2023-12-16T21:58:39.528902+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528902+0000","last_undegraded":"2023-12-16T21:58:39.528902+0000","last_fullsized":"2023-12-16T21:58:39.528902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.523268+0000","last_change":"2023-12-16T21:58:40.518810+0000","last_active":"2023-12-16T21:58:40.523268+0000","last_peered":"2023-12-16T21:58:40.523268+0000","last_clean":"2023-12-16T21:58:40.523268+0000","last_became_active":"2023-12-16T21:58:40.518226+0000","last_became_peered":"2023-12-16T21:58:40.518226+0000","last_unstale":"2023-12-16T21:58:40.523268+0000","last_undegraded":"2023-12-16T21:58:40.523268+0000","last_fullsized":"2023-12-16T21:58:40.523268+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:37:11.035240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493053+0000","last_change":"2023-12-16T21:58:40.493053+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493053+0000","last_undegraded":"2023-12-16T21:58:40.493053+0000","last_fullsized":"2023-12-16T21:58:40.493053+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477394+0000","last_change":"2023-12-16T21:58:39.477394+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477394+0000","last_undegraded":"2023-12-16T21:58:39.477394+0000","last_fullsized":"2023-12-16T21:58:39.477394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519501+0000","last_change":"2023-12-16T21:58:40.511320+0000","last_active":"2023-12-16T21:58:40.519501+0000","last_peered":"2023-12-16T21:58:40.519501+0000","last_clean":"2023-12-16T21:58:40.519501+0000","last_became_active":"2023-12-16T21:58:40.510584+0000","last_became_peered":"2023-12-16T21:58:40.510584+0000","last_unstale":"2023-12-16T21:58:40.519501+0000","last_undegraded":"2023-12-16T21:58:40.519501+0000","last_fullsized":"2023-12-16T21:58:40.519501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:36.075932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.482247+0000","last_change":"2023-12-16T21:58:39.482247+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.482247+0000","last_undegraded":"2023-12-16T21:58:39.482247+0000","last_fullsized":"2023-12-16T21:58:39.482247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.496004+0000","last_change":"2023-12-16T21:58:40.496004+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.496004+0000","last_undegraded":"2023-12-16T21:58:40.496004+0000","last_fullsized":"2023-12-16T21:58:40.496004+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477445+0000","last_change":"2023-12-16T21:58:39.477445+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477445+0000","last_undegraded":"2023-12-16T21:58:39.477445+0000","last_fullsized":"2023-12-16T21:58:39.477445+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.482543+0000","last_change":"2023-12-16T21:58:40.482543+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.482543+0000","last_undegraded":"2023-12-16T21:58:40.482543+0000","last_fullsized":"2023-12-16T21:58:40.482543+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478219+0000","last_change":"2023-12-16T21:58:39.478219+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478219+0000","last_undegraded":"2023-12-16T21:58:39.478219+0000","last_fullsized":"2023-12-16T21:58:39.478219+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.485868+0000","last_change":"2023-12-16T21:58:40.485868+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.485868+0000","last_undegraded":"2023-12-16T21:58:40.485868+0000","last_fullsized":"2023-12-16T21:58:40.485868+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.481593+0000","last_change":"2023-12-16T21:58:39.481593+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.481593+0000","last_undegraded":"2023-12-16T21:58:39.481593+0000","last_fullsized":"2023-12-16T21:58:39.481593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494957+0000","last_change":"2023-12-16T21:58:40.494957+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494957+0000","last_undegraded":"2023-12-16T21:58:40.494957+0000","last_fullsized":"2023-12-16T21:58:40.494957+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522371+0000","last_change":"2023-12-16T21:58:40.515564+0000","last_active":"2023-12-16T21:58:40.522371+0000","last_peered":"2023-12-16T21:58:40.522371+0000","last_clean":"2023-12-16T21:58:40.522371+0000","last_became_active":"2023-12-16T21:58:40.514839+0000","last_became_peered":"2023-12-16T21:58:40.514839+0000","last_unstale":"2023-12-16T21:58:40.522371+0000","last_undegraded":"2023-12-16T21:58:40.522371+0000","last_fullsized":"2023-12-16T21:58:40.522371+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:58:46.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514447+0000","last_change":"2023-12-16T21:58:40.499987+0000","last_active":"2023-12-16T21:58:40.514447+0000","last_peered":"2023-12-16T21:58:40.514447+0000","last_clean":"2023-12-16T21:58:40.514447+0000","last_became_active":"2023-12-16T21:58:40.499198+0000","last_became_peered":"2023-12-16T21:58:40.499198+0000","last_unstale":"2023-12-16T21:58:40.514447+0000","last_undegraded":"2023-12-16T21:58:40.514447+0000","last_fullsized":"2023-12-16T21:58:40.514447+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:53.492867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514335+0000","last_change":"2023-12-16T21:58:40.503566+0000","last_active":"2023-12-16T21:58:40.514335+0000","last_peered":"2023-12-16T21:58:40.514335+0000","last_clean":"2023-12-16T21:58:40.514335+0000","last_became_active":"2023-12-16T21:58:40.502798+0000","last_became_peered":"2023-12-16T21:58:40.502798+0000","last_unstale":"2023-12-16T21:58:40.514335+0000","last_undegraded":"2023-12-16T21:58:40.514335+0000","last_fullsized":"2023-12-16T21:58:40.514335+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:06:31.466515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528972+0000","last_change":"2023-12-16T21:58:39.528972+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528972+0000","last_undegraded":"2023-12-16T21:58:39.528972+0000","last_fullsized":"2023-12-16T21:58:39.528972+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.512812+0000","last_change":"2023-12-16T21:58:40.502158+0000","last_active":"2023-12-16T21:58:40.512812+0000","last_peered":"2023-12-16T21:58:40.512812+0000","last_clean":"2023-12-16T21:58:40.512812+0000","last_became_active":"2023-12-16T21:58:40.501224+0000","last_became_peered":"2023-12-16T21:58:40.501224+0000","last_unstale":"2023-12-16T21:58:40.512812+0000","last_undegraded":"2023-12-16T21:58:40.512812+0000","last_fullsized":"2023-12-16T21:58:40.512812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:11:13.641738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514394+0000","last_change":"2023-12-16T21:58:40.503347+0000","last_active":"2023-12-16T21:58:40.514394+0000","last_peered":"2023-12-16T21:58:40.514394+0000","last_clean":"2023-12-16T21:58:40.514394+0000","last_became_active":"2023-12-16T21:58:40.502397+0000","last_became_peered":"2023-12-16T21:58:40.502397+0000","last_unstale":"2023-12-16T21:58:40.514394+0000","last_undegraded":"2023-12-16T21:58:40.514394+0000","last_fullsized":"2023-12-16T21:58:40.514394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:12:26.461737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.500732+0000","last_change":"2023-12-16T21:58:40.500732+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.500732+0000","last_undegraded":"2023-12-16T21:58:40.500732+0000","last_fullsized":"2023-12-16T21:58:40.500732+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.523302+0000","last_change":"2023-12-16T21:58:40.519887+0000","last_active":"2023-12-16T21:58:40.523302+0000","last_peered":"2023-12-16T21:58:40.523302+0000","last_clean":"2023-12-16T21:58:40.523302+0000","last_became_active":"2023-12-16T21:58:40.518760+0000","last_became_peered":"2023-12-16T21:58:40.518760+0000","last_unstale":"2023-12-16T21:58:40.523302+0000","last_undegraded":"2023-12-16T21:58:40.523302+0000","last_fullsized":"2023-12-16T21:58:40.523302+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:18:57.185826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480909+0000","last_change":"2023-12-16T21:58:39.480909+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480909+0000","last_undegraded":"2023-12-16T21:58:39.480909+0000","last_fullsized":"2023-12-16T21:58:39.480909+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494047+0000","last_change":"2023-12-16T21:58:40.494047+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494047+0000","last_undegraded":"2023-12-16T21:58:40.494047+0000","last_fullsized":"2023-12-16T21:58:40.494047+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528116+0000","last_change":"2023-12-16T21:58:39.528116+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528116+0000","last_undegraded":"2023-12-16T21:58:39.528116+0000","last_fullsized":"2023-12-16T21:58:39.528116+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480710+0000","last_change":"2023-12-16T21:58:39.480710+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480710+0000","last_undegraded":"2023-12-16T21:58:39.480710+0000","last_fullsized":"2023-12-16T21:58:39.480710+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483887+0000","last_change":"2023-12-16T21:58:40.483887+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483887+0000","last_undegraded":"2023-12-16T21:58:40.483887+0000","last_fullsized":"2023-12-16T21:58:40.483887+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528182+0000","last_change":"2023-12-16T21:58:39.528182+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528182+0000","last_undegraded":"2023-12-16T21:58:39.528182+0000","last_fullsized":"2023-12-16T21:58:39.528182+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480584+0000","last_change":"2023-12-16T21:58:39.480584+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480584+0000","last_undegraded":"2023-12-16T21:58:39.480584+0000","last_fullsized":"2023-12-16T21:58:39.480584+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522536+0000","last_change":"2023-12-16T21:58:40.516094+0000","last_active":"2023-12-16T21:58:40.522536+0000","last_peered":"2023-12-16T21:58:40.522536+0000","last_clean":"2023-12-16T21:58:40.522536+0000","last_became_active":"2023-12-16T21:58:40.515413+0000","last_became_peered":"2023-12-16T21:58:40.515413+0000","last_unstale":"2023-12-16T21:58:40.522536+0000","last_undegraded":"2023-12-16T21:58:40.522536+0000","last_fullsized":"2023-12-16T21:58:40.522536+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:48:35.936102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.484418+0000","last_change":"2023-12-16T21:58:40.484418+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.484418+0000","last_undegraded":"2023-12-16T21:58:40.484418+0000","last_fullsized":"2023-12-16T21:58:40.484418+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477392+0000","last_change":"2023-12-16T21:58:39.477392+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477392+0000","last_undegraded":"2023-12-16T21:58:39.477392+0000","last_fullsized":"2023-12-16T21:58:39.477392+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.497157+0000","last_change":"2023-12-16T21:58:40.497157+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.497157+0000","last_undegraded":"2023-12-16T21:58:40.497157+0000","last_fullsized":"2023-12-16T21:58:40.497157+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519500+0000","last_change":"2023-12-16T21:58:40.516743+0000","last_active":"2023-12-16T21:58:40.519500+0000","last_peered":"2023-12-16T21:58:40.519500+0000","last_clean":"2023-12-16T21:58:40.519500+0000","last_became_active":"2023-12-16T21:58:40.515578+0000","last_became_peered":"2023-12-16T21:58:40.515578+0000","last_unstale":"2023-12-16T21:58:40.519500+0000","last_undegraded":"2023-12-16T21:58:40.519500+0000","last_fullsized":"2023-12-16T21:58:40.519500+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:18:32.213979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:58:39.484154+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:58:39.484154+0000","last_peered":"2023-12-16T21:58:39.484154+0000","last_clean":"2023-12-16T21:58:39.484154+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:58:39.484154+0000","last_undegraded":"2023-12-16T21:58:39.484154+0000","last_fullsized":"2023-12-16T21:58:39.484154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":68,"acting":68,"num_store_stats":4},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":732948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.1779999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.26}]},{"osd":4,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.81599999999999995}]},{"osd":5,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.0620000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.109}]}]},{"osd":6,"up_from":16,"seq":68719476753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.66800000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.71899999999999997}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.754}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":1.083}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":5.5620000000000003,"5min":5.5620000000000003,"15min":5.5620000000000003},"last":1.0549999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.93200000000000005}]}]},{"osd":5,"up_from":16,"seq":68719476754,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.56200000000000006}]},{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.72999999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.0569999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.98399999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.018}]},{"osd":6,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.254,"5min":3.254,"15min":3.254},"last":0.95199999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.512,"5min":2.512,"15min":2.512},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":1.339}]}]},{"osd":4,"up_from":16,"seq":68719476752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.79100000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.91600000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.998},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.95599999999999996}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.123}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.208},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.173}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.085}]}]},{"osd":3,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.76700000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.80700000000000005}]}]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.371},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.435},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.64300000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.65900000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.87}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.70599999999999996}]}]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":942080,"data_stored":720864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.79700000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.85599999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.21},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.046}]},{"osd":5,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":1.327},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.2569999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.081,"5min":2.081,"15min":2.081},"last":1.1719999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.3779999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542161,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31528,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":348160,"data_stored":130496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.63},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.45800000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.503}]},{"osd":3,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.56699999999999995}]},{"osd":4,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":0.70299999999999996}]},{"osd":6,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.0660000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:58:44.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:58:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.133+0000 7f55e2111640 1 Processor -- start 2023-12-16T21:58:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55e2111640 1 -- start start 2023-12-16T21:58:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55e2111640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55dc06cdb0 0x7f55dc150250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55e2111640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc070460 con 0x7f55dc06d190 2023-12-16T21:58:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55e2111640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc15ab20 con 0x7f55dc15a4b0 2023-12-16T21:58:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55e2111640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc15a900 con 0x7f55dc06cdb0 2023-12-16T21:58:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55daffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f55dc06d190 0x7f55dc159da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57392/0 (socket says 172.21.15.28:57392) 2023-12-16T21:58:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55daffd640 1 -- 172.21.15.28:0/1880313375 learned_addr learned my addr 172.21.15.28:0/1880313375 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55db7fe640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55dc06cdb0 0x7f55dc150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676633185 0 0) 0x7f55dc070460 con 0x7f55dc06d190 2023-12-16T21:58:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c4003530 con 0x7f55dc06d190 2023-12-16T21:58:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 629063805 0 0) 0x7f55c4003530 con 0x7f55dc06d190 2023-12-16T21:58:45.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55dc06cdb0 msgr2=0x7f55dc150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:45.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55dc06cdb0 0x7f55dc150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 >> v1:172.21.15.122:6789/0 conn(0x7f55dc15a4b0 legacy=0x7f55dc0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc070690 con 0x7f55dc06d190 2023-12-16T21:58:45.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f55d0001680 con 0x7f55dc06d190 2023-12-16T21:58:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 >> v1:172.21.15.28:6789/0 conn(0x7f55dc06d190 legacy=0x7f55dc159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c40039b0 0x7f55c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c40047d0 0x7f55c4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55dc06cdb0 0x7f55c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc15a900 con 0x7f55c40047d0 2023-12-16T21:58:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c400bac0 con 0x7f55c40039b0 2023-12-16T21:58:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c400de90 con 0x7f55dc06cdb0 2023-12-16T21:58:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55daffd640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c40047d0 0x7f55c4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55db7fe640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c40039b0 0x7f55c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55dbfff640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55dc06cdb0 0x7f55c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55daffd640 1 -- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55dc06cdb0 msgr2=0x7f55c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55daffd640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55dc06cdb0 0x7f55c40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55daffd640 1 -- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c40039b0 msgr2=0x7f55c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55daffd640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55c40039b0 0x7f55c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.137+0000 7f55daffd640 1 -- 172.21.15.28:0/1880313375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c4003530 con 0x7f55c40047d0 2023-12-16T21:58:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55daffd640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c40047d0 0x7f55c4004c50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f55d000c240 tx=0x7f55d001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70dcb1e58ed598da server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d0005ae0 con 0x7f55c40047d0 2023-12-16T21:58:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d0005c40 con 0x7f55c40047d0 2023-12-16T21:58:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55da7fc640 1 -- 172.21.15.28:0/1880313375 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d0007070 con 0x7f55c40047d0 2023-12-16T21:58:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- 172.21.15.28:0/1880313375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c40047d0 msgr2=0x7f55c4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 --2- 172.21.15.28:0/1880313375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c40047d0 0x7f55c4004c50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f55d000c240 tx=0x7f55d001cc40 comp rx=0 tx=0).stop 2023-12-16T21:58:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55dbfff640 1 -- 172.21.15.28:0/1880313375 reap_dead start 2023-12-16T21:58:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- 172.21.15.28:0/1880313375 shutdown_connections 2023-12-16T21:58:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- 172.21.15.28:0/1880313375 >> 172.21.15.28:0/1880313375 conn(0x7f55dc06a530 msgr2=0x7f55dc06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- 172.21.15.28:0/1880313375 shutdown_connections 2023-12-16T21:58:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- 172.21.15.28:0/1880313375 wait complete. 2023-12-16T21:58:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 Processor -- start 2023-12-16T21:58:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- start start 2023-12-16T21:58:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55c40047d0 0x7f55dc141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55dc145370 0x7f55dc1f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cd1c0 0x7f55dc0cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0cdb80 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc13f370 con 0x7f55dc145370 2023-12-16T21:58:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55e2111640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc1457f0 con 0x7f55c40047d0 2023-12-16T21:58:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cd1c0 0x7f55dc0cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cd1c0 0x7f55dc0cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36974/0 (socket says 172.21.15.28:36974) 2023-12-16T21:58:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55dbfff640 1 -- 172.21.15.28:0/2122684765 learned_addr learned my addr 172.21.15.28:0/2122684765 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55db7fe640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55c40047d0 0x7f55dc141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.141+0000 7f55daffd640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55dc145370 0x7f55dc1f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55dbfff640 1 -- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55c40047d0 msgr2=0x7f55dc141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55dbfff640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55c40047d0 0x7f55dc141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55dbfff640 1 -- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55dc145370 msgr2=0x7f55dc1f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55dbfff640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55dc145370 0x7f55dc1f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55dbfff640 1 -- 172.21.15.28:0/2122684765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc141940 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55dbfff640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cd1c0 0x7f55dc0cd640 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f55dc15a890 tx=0x7f55cc00e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55c3fff640 1 -- 172.21.15.28:0/2122684765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0052e0 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55c3fff640 1 -- 172.21.15.28:0/2122684765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc005460 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc141c20 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55c3fff640 1 -- 172.21.15.28:0/2122684765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc003840 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc1421e0 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc13f500 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55c3fff640 1 -- 172.21.15.28:0/2122684765 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f55cc0039a0 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55c3fff640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f55c80664a0 0x7f55c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55c3fff640 1 -- 172.21.15.28:0/2122684765 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55cc0834c0 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55db7fe640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f55c80664a0 0x7f55c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.162 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.145+0000 7f55db7fe640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f55c80664a0 0x7f55c8068960 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f55d4003c10 tx=0x7f55d400b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.162 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.149+0000 7f55c3fff640 1 -- 172.21.15.28:0/2122684765 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f55cc0509f0 con 0x7f55dc0cd1c0 2023-12-16T21:58:45.308 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.301+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55dc150250 con 0x7f55c80664a0 2023-12-16T21:58:45.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.325+0000 7f55c3fff640 1 -- 172.21.15.28:0/2122684765 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278874 (secure 0 0 0) 0x7f55d400f040 con 0x7f55c80664a0 2023-12-16T21:58:45.333 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:58:45.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f55c80664a0 msgr2=0x7f55c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f55c80664a0 0x7f55c8068960 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f55d4003c10 tx=0x7f55d400b040 comp rx=0 tx=0).stop 2023-12-16T21:58:45.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cd1c0 msgr2=0x7f55dc0cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cd1c0 0x7f55dc0cd640 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f55dc15a890 tx=0x7f55cc00e8a0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 shutdown_connections 2023-12-16T21:58:45.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f55c80664a0 0x7f55c8068960 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cd1c0 0x7f55dc0cd640 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55dc145370 0x7f55dc1f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 --2- 172.21.15.28:0/2122684765 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f55c40047d0 0x7f55dc141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 >> 172.21.15.28:0/2122684765 conn(0x7f55dc06a530 msgr2=0x7f55dc15c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:45.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 shutdown_connections 2023-12-16T21:58:45.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.329+0000 7f55e2111640 1 -- 172.21.15.28:0/2122684765 wait complete. 2023-12-16T21:58:45.351 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2023-12-16T21:58:43.402199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":132,"acting":132,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":240992,"kb_used_data":4000,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733728,"statfs":{"total":773094113280,"available":772847337472,"internally_reserved":0,"allocated":4096000,"data_stored":2284674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001090"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483829+0000","last_change":"2023-12-16T21:58:40.483829+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483829+0000","last_undegraded":"2023-12-16T21:58:40.483829+0000","last_fullsized":"2023-12-16T21:58:40.483829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.492190+0000","last_change":"2023-12-16T21:58:40.492190+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.492190+0000","last_undegraded":"2023-12-16T21:58:40.492190+0000","last_fullsized":"2023-12-16T21:58:40.492190+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.499126+0000","last_change":"2023-12-16T21:58:40.499126+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.499126+0000","last_undegraded":"2023-12-16T21:58:40.499126+0000","last_fullsized":"2023-12-16T21:58:40.499126+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.505046+0000","last_change":"2023-12-16T21:58:40.505046+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.505046+0000","last_undegraded":"2023-12-16T21:58:40.505046+0000","last_fullsized":"2023-12-16T21:58:40.505046+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.488032+0000","last_change":"2023-12-16T21:58:40.488032+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.488032+0000","last_undegraded":"2023-12-16T21:58:40.488032+0000","last_fullsized":"2023-12-16T21:58:40.488032+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494042+0000","last_change":"2023-12-16T21:58:40.494042+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494042+0000","last_undegraded":"2023-12-16T21:58:40.494042+0000","last_fullsized":"2023-12-16T21:58:40.494042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.492852+0000","last_change":"2023-12-16T21:58:40.492852+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.492852+0000","last_undegraded":"2023-12-16T21:58:40.492852+0000","last_fullsized":"2023-12-16T21:58:40.492852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.496638+0000","last_change":"2023-12-16T21:58:40.496638+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.496638+0000","last_undegraded":"2023-12-16T21:58:40.496638+0000","last_fullsized":"2023-12-16T21:58:40.496638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493692+0000","last_change":"2023-12-16T21:58:40.493692+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493692+0000","last_undegraded":"2023-12-16T21:58:40.493692+0000","last_fullsized":"2023-12-16T21:58:40.493692+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.497617+0000","last_change":"2023-12-16T21:58:40.497617+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.497617+0000","last_undegraded":"2023-12-16T21:58:40.497617+0000","last_fullsized":"2023-12-16T21:58:40.497617+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491447+0000","last_change":"2023-12-16T21:58:40.491447+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491447+0000","last_undegraded":"2023-12-16T21:58:40.491447+0000","last_fullsized":"2023-12-16T21:58:40.491447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.487840+0000","last_change":"2023-12-16T21:58:40.487840+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.487840+0000","last_undegraded":"2023-12-16T21:58:40.487840+0000","last_fullsized":"2023-12-16T21:58:40.487840+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.490713+0000","last_change":"2023-12-16T21:58:40.490713+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.490713+0000","last_undegraded":"2023-12-16T21:58:40.490713+0000","last_fullsized":"2023-12-16T21:58:40.490713+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491975+0000","last_change":"2023-12-16T21:58:40.491975+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491975+0000","last_undegraded":"2023-12-16T21:58:40.491975+0000","last_fullsized":"2023-12-16T21:58:40.491975+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.488011+0000","last_change":"2023-12-16T21:58:40.488011+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.488011+0000","last_undegraded":"2023-12-16T21:58:40.488011+0000","last_fullsized":"2023-12-16T21:58:40.488011+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.495037+0000","last_change":"2023-12-16T21:58:40.495037+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.495037+0000","last_undegraded":"2023-12-16T21:58:40.495037+0000","last_fullsized":"2023-12-16T21:58:40.495037+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.484902+0000","last_change":"2023-12-16T21:58:40.484902+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.484902+0000","last_undegraded":"2023-12-16T21:58:40.484902+0000","last_fullsized":"2023-12-16T21:58:40.484902+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.501035+0000","last_change":"2023-12-16T21:58:40.501035+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.501035+0000","last_undegraded":"2023-12-16T21:58:40.501035+0000","last_fullsized":"2023-12-16T21:58:40.501035+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.517957+0000","last_change":"2023-12-16T21:58:40.510463+0000","last_active":"2023-12-16T21:58:40.517957+0000","last_peered":"2023-12-16T21:58:40.517957+0000","last_clean":"2023-12-16T21:58:40.517957+0000","last_became_active":"2023-12-16T21:58:40.509883+0000","last_became_peered":"2023-12-16T21:58:40.509883+0000","last_unstale":"2023-12-16T21:58:40.517957+0000","last_undegraded":"2023-12-16T21:58:40.517957+0000","last_fullsized":"2023-12-16T21:58:40.517957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:46:52.112074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.482640+0000","last_change":"2023-12-16T21:58:40.482640+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.482640+0000","last_undegraded":"2023-12-16T21:58:40.482640+0000","last_fullsized":"2023-12-16T21:58:40.482640+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.529438+0000","last_change":"2023-12-16T21:58:39.529438+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.529438+0000","last_undegraded":"2023-12-16T21:58:39.529438+0000","last_fullsized":"2023-12-16T21:58:39.529438+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.489570+0000","last_change":"2023-12-16T21:58:40.489570+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.489570+0000","last_undegraded":"2023-12-16T21:58:40.489570+0000","last_fullsized":"2023-12-16T21:58:40.489570+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478861+0000","last_change":"2023-12-16T21:58:39.478861+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478861+0000","last_undegraded":"2023-12-16T21:58:39.478861+0000","last_fullsized":"2023-12-16T21:58:39.478861+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:39.467225+0000","last_change":"2023-12-16T21:58:39.467225+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.467225+0000","last_undegraded":"2023-12-16T21:58:39.467225+0000","last_fullsized":"2023-12-16T21:58:39.467225+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493858+0000","last_change":"2023-12-16T21:58:40.493858+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493858+0000","last_undegraded":"2023-12-16T21:58:40.493858+0000","last_fullsized":"2023-12-16T21:58:40.493858+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522509+0000","last_change":"2023-12-16T21:58:40.516801+0000","last_active":"2023-12-16T21:58:40.522509+0000","last_peered":"2023-12-16T21:58:40.522509+0000","last_clean":"2023-12-16T21:58:40.522509+0000","last_became_active":"2023-12-16T21:58:40.515829+0000","last_became_peered":"2023-12-16T21:58:40.515829+0000","last_unstale":"2023-12-16T21:58:40.522509+0000","last_undegraded":"2023-12-16T21:58:40.522509+0000","last_fullsized":"2023-12-16T21:58:40.522509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:26:05.935472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494574+0000","last_change":"2023-12-16T21:58:40.494574+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494574+0000","last_undegraded":"2023-12-16T21:58:40.494574+0000","last_fullsized":"2023-12-16T21:58:40.494574+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514237+0000","last_change":"2023-12-16T21:58:40.501591+0000","last_active":"2023-12-16T21:58:40.514237+0000","last_peered":"2023-12-16T21:58:40.514237+0000","last_clean":"2023-12-16T21:58:40.514237+0000","last_became_active":"2023-12-16T21:58:40.500870+0000","last_became_peered":"2023-12-16T21:58:40.500870+0000","last_unstale":"2023-12-16T21:58:40.514237+0000","last_undegraded":"2023-12-16T21:58:40.514237+0000","last_fullsized":"2023-12-16T21:58:40.514237+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:34:30.857548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483333+0000","last_change":"2023-12-16T21:58:40.483333+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483333+0000","last_undegraded":"2023-12-16T21:58:40.483333+0000","last_fullsized":"2023-12-16T21:58:40.483333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478762+0000","last_change":"2023-12-16T21:58:39.478762+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478762+0000","last_undegraded":"2023-12-16T21:58:39.478762+0000","last_fullsized":"2023-12-16T21:58:39.478762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519509+0000","last_change":"2023-12-16T21:58:40.511738+0000","last_active":"2023-12-16T21:58:40.519509+0000","last_peered":"2023-12-16T21:58:40.519509+0000","last_clean":"2023-12-16T21:58:40.519509+0000","last_became_active":"2023-12-16T21:58:40.510752+0000","last_became_peered":"2023-12-16T21:58:40.510752+0000","last_unstale":"2023-12-16T21:58:40.519509+0000","last_undegraded":"2023-12-16T21:58:40.519509+0000","last_fullsized":"2023-12-16T21:58:40.519509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:47:01.519007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491282+0000","last_change":"2023-12-16T21:58:40.491282+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491282+0000","last_undegraded":"2023-12-16T21:58:40.491282+0000","last_fullsized":"2023-12-16T21:58:40.491282+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477459+0000","last_change":"2023-12-16T21:58:39.477459+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477459+0000","last_undegraded":"2023-12-16T21:58:39.477459+0000","last_fullsized":"2023-12-16T21:58:39.477459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528902+0000","last_change":"2023-12-16T21:58:39.528902+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528902+0000","last_undegraded":"2023-12-16T21:58:39.528902+0000","last_fullsized":"2023-12-16T21:58:39.528902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.523268+0000","last_change":"2023-12-16T21:58:40.518810+0000","last_active":"2023-12-16T21:58:40.523268+0000","last_peered":"2023-12-16T21:58:40.523268+0000","last_clean":"2023-12-16T21:58:40.523268+0000","last_became_active":"2023-12-16T21:58:40.518226+0000","last_became_peered":"2023-12-16T21:58:40.518226+0000","last_unstale":"2023-12-16T21:58:40.523268+0000","last_undegraded":"2023-12-16T21:58:40.523268+0000","last_fullsized":"2023-12-16T21:58:40.523268+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:37:11.035240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493053+0000","last_change":"2023-12-16T21:58:40.493053+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493053+0000","last_undegraded":"2023-12-16T21:58:40.493053+0000","last_fullsized":"2023-12-16T21:58:40.493053+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477394+0000","last_change":"2023-12-16T21:58:39.477394+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477394+0000","last_undegraded":"2023-12-16T21:58:39.477394+0000","last_fullsized":"2023-12-16T21:58:39.477394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519501+0000","last_change":"2023-12-16T21:58:40.511320+0000","last_active":"2023-12-16T21:58:40.519501+0000","last_peered":"2023-12-16T21:58:40.519501+0000","last_clean":"2023-12-16T21:58:40.519501+0000","last_became_active":"2023-12-16T21:58:40.510584+0000","last_became_peered":"2023-12-16T21:58:40.510584+0000","last_unstale":"2023-12-16T21:58:40.519501+0000","last_undegraded":"2023-12-16T21:58:40.519501+0000","last_fullsized":"2023-12-16T21:58:40.519501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:36.075932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.482247+0000","last_change":"2023-12-16T21:58:39.482247+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.482247+0000","last_undegraded":"2023-12-16T21:58:39.482247+0000","last_fullsized":"2023-12-16T21:58:39.482247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.496004+0000","last_change":"2023-12-16T21:58:40.496004+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.496004+0000","last_undegraded":"2023-12-16T21:58:40.496004+0000","last_fullsized":"2023-12-16T21:58:40.496004+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477445+0000","last_change":"2023-12-16T21:58:39.477445+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477445+0000","last_undegraded":"2023-12-16T21:58:39.477445+0000","last_fullsized":"2023-12-16T21:58:39.477445+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.482543+0000","last_change":"2023-12-16T21:58:40.482543+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.482543+0000","last_undegraded":"2023-12-16T21:58:40.482543+0000","last_fullsized":"2023-12-16T21:58:40.482543+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478219+0000","last_change":"2023-12-16T21:58:39.478219+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478219+0000","last_undegraded":"2023-12-16T21:58:39.478219+0000","last_fullsized":"2023-12-16T21:58:39.478219+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.485868+0000","last_change":"2023-12-16T21:58:40.485868+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.485868+0000","last_undegraded":"2023-12-16T21:58:40.485868+0000","last_fullsized":"2023-12-16T21:58:40.485868+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.481593+0000","last_change":"2023-12-16T21:58:39.481593+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.481593+0000","last_undegraded":"2023-12-16T21:58:39.481593+0000","last_fullsized":"2023-12-16T21:58:39.481593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494957+0000","last_change":"2023-12-16T21:58:40.494957+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494957+0000","last_undegraded":"2023-12-16T21:58:40.494957+0000","last_fullsized":"2023-12-16T21:58:40.494957+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522371+0000","last_change":"2023-12-16T21:58:40.515564+0000","last_active":"2023-12-16T21:58:40.522371+0000","last_peered":"2023-12-16T21:58:40.522371+0000","last_clean":"2023-12-16T21:58:40.522371+0000","last_became_active":"2023-12-16T21:58:40.514839+0000","last_became_peered":"2023-12-16T21:58:40.514839+0000","last_unstale":"2023-12-16T21:58:40.522371+0000","last_undegraded":"2023-12-16T21:58:40.522371+0000","last_fullsized":"2023-12-16T21:58:40.522371+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:58:46.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514447+0000","last_change":"2023-12-16T21:58:40.499987+0000","last_active":"2023-12-16T21:58:40.514447+0000","last_peered":"2023-12-16T21:58:40.514447+0000","last_clean":"2023-12-16T21:58:40.514447+0000","last_became_active":"2023-12-16T21:58:40.499198+0000","last_became_peered":"2023-12-16T21:58:40.499198+0000","last_unstale":"2023-12-16T21:58:40.514447+0000","last_undegraded":"2023-12-16T21:58:40.514447+0000","last_fullsized":"2023-12-16T21:58:40.514447+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:53.492867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514335+0000","last_change":"2023-12-16T21:58:40.503566+0000","last_active":"2023-12-16T21:58:40.514335+0000","last_peered":"2023-12-16T21:58:40.514335+0000","last_clean":"2023-12-16T21:58:40.514335+0000","last_became_active":"2023-12-16T21:58:40.502798+0000","last_became_peered":"2023-12-16T21:58:40.502798+0000","last_unstale":"2023-12-16T21:58:40.514335+0000","last_undegraded":"2023-12-16T21:58:40.514335+0000","last_fullsized":"2023-12-16T21:58:40.514335+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:06:31.466515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528972+0000","last_change":"2023-12-16T21:58:39.528972+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528972+0000","last_undegraded":"2023-12-16T21:58:39.528972+0000","last_fullsized":"2023-12-16T21:58:39.528972+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.512812+0000","last_change":"2023-12-16T21:58:40.502158+0000","last_active":"2023-12-16T21:58:40.512812+0000","last_peered":"2023-12-16T21:58:40.512812+0000","last_clean":"2023-12-16T21:58:40.512812+0000","last_became_active":"2023-12-16T21:58:40.501224+0000","last_became_peered":"2023-12-16T21:58:40.501224+0000","last_unstale":"2023-12-16T21:58:40.512812+0000","last_undegraded":"2023-12-16T21:58:40.512812+0000","last_fullsized":"2023-12-16T21:58:40.512812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:11:13.641738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514394+0000","last_change":"2023-12-16T21:58:40.503347+0000","last_active":"2023-12-16T21:58:40.514394+0000","last_peered":"2023-12-16T21:58:40.514394+0000","last_clean":"2023-12-16T21:58:40.514394+0000","last_became_active":"2023-12-16T21:58:40.502397+0000","last_became_peered":"2023-12-16T21:58:40.502397+0000","last_unstale":"2023-12-16T21:58:40.514394+0000","last_undegraded":"2023-12-16T21:58:40.514394+0000","last_fullsized":"2023-12-16T21:58:40.514394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:12:26.461737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.500732+0000","last_change":"2023-12-16T21:58:40.500732+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.500732+0000","last_undegraded":"2023-12-16T21:58:40.500732+0000","last_fullsized":"2023-12-16T21:58:40.500732+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.523302+0000","last_change":"2023-12-16T21:58:40.519887+0000","last_active":"2023-12-16T21:58:40.523302+0000","last_peered":"2023-12-16T21:58:40.523302+0000","last_clean":"2023-12-16T21:58:40.523302+0000","last_became_active":"2023-12-16T21:58:40.518760+0000","last_became_peered":"2023-12-16T21:58:40.518760+0000","last_unstale":"2023-12-16T21:58:40.523302+0000","last_undegraded":"2023-12-16T21:58:40.523302+0000","last_fullsized":"2023-12-16T21:58:40.523302+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:18:57.185826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480909+0000","last_change":"2023-12-16T21:58:39.480909+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480909+0000","last_undegraded":"2023-12-16T21:58:39.480909+0000","last_fullsized":"2023-12-16T21:58:39.480909+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494047+0000","last_change":"2023-12-16T21:58:40.494047+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494047+0000","last_undegraded":"2023-12-16T21:58:40.494047+0000","last_fullsized":"2023-12-16T21:58:40.494047+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528116+0000","last_change":"2023-12-16T21:58:39.528116+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528116+0000","last_undegraded":"2023-12-16T21:58:39.528116+0000","last_fullsized":"2023-12-16T21:58:39.528116+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480710+0000","last_change":"2023-12-16T21:58:39.480710+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480710+0000","last_undegraded":"2023-12-16T21:58:39.480710+0000","last_fullsized":"2023-12-16T21:58:39.480710+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483887+0000","last_change":"2023-12-16T21:58:40.483887+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483887+0000","last_undegraded":"2023-12-16T21:58:40.483887+0000","last_fullsized":"2023-12-16T21:58:40.483887+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528182+0000","last_change":"2023-12-16T21:58:39.528182+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528182+0000","last_undegraded":"2023-12-16T21:58:39.528182+0000","last_fullsized":"2023-12-16T21:58:39.528182+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480584+0000","last_change":"2023-12-16T21:58:39.480584+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480584+0000","last_undegraded":"2023-12-16T21:58:39.480584+0000","last_fullsized":"2023-12-16T21:58:39.480584+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522536+0000","last_change":"2023-12-16T21:58:40.516094+0000","last_active":"2023-12-16T21:58:40.522536+0000","last_peered":"2023-12-16T21:58:40.522536+0000","last_clean":"2023-12-16T21:58:40.522536+0000","last_became_active":"2023-12-16T21:58:40.515413+0000","last_became_peered":"2023-12-16T21:58:40.515413+0000","last_unstale":"2023-12-16T21:58:40.522536+0000","last_undegraded":"2023-12-16T21:58:40.522536+0000","last_fullsized":"2023-12-16T21:58:40.522536+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:48:35.936102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.484418+0000","last_change":"2023-12-16T21:58:40.484418+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.484418+0000","last_undegraded":"2023-12-16T21:58:40.484418+0000","last_fullsized":"2023-12-16T21:58:40.484418+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477392+0000","last_change":"2023-12-16T21:58:39.477392+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477392+0000","last_undegraded":"2023-12-16T21:58:39.477392+0000","last_fullsized":"2023-12-16T21:58:39.477392+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.497157+0000","last_change":"2023-12-16T21:58:40.497157+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.497157+0000","last_undegraded":"2023-12-16T21:58:40.497157+0000","last_fullsized":"2023-12-16T21:58:40.497157+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519500+0000","last_change":"2023-12-16T21:58:40.516743+0000","last_active":"2023-12-16T21:58:40.519500+0000","last_peered":"2023-12-16T21:58:40.519500+0000","last_clean":"2023-12-16T21:58:40.519500+0000","last_became_active":"2023-12-16T21:58:40.515578+0000","last_became_peered":"2023-12-16T21:58:40.515578+0000","last_unstale":"2023-12-16T21:58:40.519500+0000","last_undegraded":"2023-12-16T21:58:40.519500+0000","last_fullsized":"2023-12-16T21:58:40.519500+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:18:32.213979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:58:39.484154+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:58:39.484154+0000","last_peered":"2023-12-16T21:58:39.484154+0000","last_clean":"2023-12-16T21:58:39.484154+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:58:39.484154+0000","last_undegraded":"2023-12-16T21:58:39.484154+0000","last_fullsized":"2023-12-16T21:58:39.484154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":68,"acting":68,"num_store_stats":4},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":62,"acting":62,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":732948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.1779999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.26}]},{"osd":4,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.81599999999999995}]},{"osd":5,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.0620000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.109}]}]},{"osd":6,"up_from":16,"seq":68719476753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.66800000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.71899999999999997}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.754}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":1.083}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":5.5620000000000003,"5min":5.5620000000000003,"15min":5.5620000000000003},"last":1.0549999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.93200000000000005}]}]},{"osd":5,"up_from":16,"seq":68719476754,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.56200000000000006}]},{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.72999999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.0569999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.98399999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.018}]},{"osd":6,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.254,"5min":3.254,"15min":3.254},"last":0.95199999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.512,"5min":2.512,"15min":2.512},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":1.339}]}]},{"osd":4,"up_from":16,"seq":68719476752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.79100000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.91600000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.998},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.95599999999999996}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.123}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.208},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.173}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.085}]}]},{"osd":3,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.76700000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.80700000000000005}]}]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.371},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.435},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.64300000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.65900000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.87}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.70599999999999996}]}]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":942080,"data_stored":720864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.79700000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.85599999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.21},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.046}]},{"osd":5,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":1.327},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.2569999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.081,"5min":2.081,"15min":2.081},"last":1.1719999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.3779999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542161,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31528,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":348160,"data_stored":130496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.63},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.45800000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.503}]},{"osd":3,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.56699999999999995}]},{"osd":4,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":0.70299999999999996}]},{"osd":6,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":1.0660000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.483829+0000', 'last_change': '2023-12-16T21:58:40.483829+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.483829+0000', 'last_undegraded': '2023-12-16T21:58:40.483829+0000', 'last_fullsized': '2023-12-16T21:58:40.483829+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.492190+0000', 'last_change': '2023-12-16T21:58:40.492190+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.492190+0000', 'last_undegraded': '2023-12-16T21:58:40.492190+0000', 'last_fullsized': '2023-12-16T21:58:40.492190+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2023-12-16T21:58:45.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2023-12-16T21:58:45.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2023-12-16T21:58:45.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.499126+0000', 'last_change': '2023-12-16T21:58:40.499126+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.499126+0000', 'last_undegraded': '2023-12-16T21:58:40.499126+0000', 'last_fullsized': '2023-12-16T21:58:40.499126+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2023-12-16T21:58:45.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.505046+0000', 'last_change': '2023-12-16T21:58:40.505046+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.505046+0000', 'last_undegraded': '2023-12-16T21:58:40.505046+0000', 'last_fullsized': '2023-12-16T21:58:40.505046+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2023-12-16T21:58:45.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.488032+0000', 'last_change': '2023-12-16T21:58:40.488032+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.488032+0000', 'last_undegraded': '2023-12-16T21:58:40.488032+0000', 'last_fullsized': '2023-12-16T21:58:40.488032+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-16T21:58:45.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2023-12-16T21:58:45.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2023-12-16T21:58:45.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.494042+0000', 'last_change': '2023-12-16T21:58:40.494042+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.494042+0000', 'last_undegraded': '2023-12-16T21:58:40.494042+0000', 'last_fullsized': '2023-12-16T21:58:40.494042+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2023-12-16T21:58:45.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.492852+0000', 'last_change': '2023-12-16T21:58:40.492852+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.492852+0000', 'last_undegraded': '2023-12-16T21:58:40.492852+0000', 'last_fullsized': '2023-12-16T21:58:40.492852+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2023-12-16T21:58:45.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2023-12-16T21:58:45.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2023-12-16T21:58:45.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.496638+0000', 'last_change': '2023-12-16T21:58:40.496638+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.496638+0000', 'last_undegraded': '2023-12-16T21:58:40.496638+0000', 'last_fullsized': '2023-12-16T21:58:40.496638+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2023-12-16T21:58:45.361 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.493692+0000', 'last_change': '2023-12-16T21:58:40.493692+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.493692+0000', 'last_undegraded': '2023-12-16T21:58:40.493692+0000', 'last_fullsized': '2023-12-16T21:58:40.493692+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.361 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2023-12-16T21:58:45.361 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.497617+0000', 'last_change': '2023-12-16T21:58:40.497617+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.497617+0000', 'last_undegraded': '2023-12-16T21:58:40.497617+0000', 'last_fullsized': '2023-12-16T21:58:40.497617+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.361 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2023-12-16T21:58:45.361 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.362 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2023-12-16T21:58:45.362 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.362 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2023-12-16T21:58:45.362 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2023-12-16T21:58:45.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.491447+0000', 'last_change': '2023-12-16T21:58:40.491447+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.491447+0000', 'last_undegraded': '2023-12-16T21:58:40.491447+0000', 'last_fullsized': '2023-12-16T21:58:40.491447+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-16T21:58:45.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2023-12-16T21:58:45.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.487840+0000', 'last_change': '2023-12-16T21:58:40.487840+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.487840+0000', 'last_undegraded': '2023-12-16T21:58:40.487840+0000', 'last_fullsized': '2023-12-16T21:58:40.487840+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-16T21:58:45.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2023-12-16T21:58:45.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2023-12-16T21:58:45.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.490713+0000', 'last_change': '2023-12-16T21:58:40.490713+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.490713+0000', 'last_undegraded': '2023-12-16T21:58:40.490713+0000', 'last_fullsized': '2023-12-16T21:58:40.490713+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2023-12-16T21:58:45.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2023-12-16T21:58:45.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2023-12-16T21:58:45.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2023-12-16T21:58:45.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.491975+0000', 'last_change': '2023-12-16T21:58:40.491975+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.491975+0000', 'last_undegraded': '2023-12-16T21:58:40.491975+0000', 'last_fullsized': '2023-12-16T21:58:40.491975+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2023-12-16T21:58:45.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.488011+0000', 'last_change': '2023-12-16T21:58:40.488011+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.488011+0000', 'last_undegraded': '2023-12-16T21:58:40.488011+0000', 'last_fullsized': '2023-12-16T21:58:40.488011+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-16T21:58:45.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2023-12-16T21:58:45.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.495037+0000', 'last_change': '2023-12-16T21:58:40.495037+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.495037+0000', 'last_undegraded': '2023-12-16T21:58:40.495037+0000', 'last_fullsized': '2023-12-16T21:58:40.495037+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2023-12-16T21:58:45.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.484902+0000', 'last_change': '2023-12-16T21:58:40.484902+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.484902+0000', 'last_undegraded': '2023-12-16T21:58:40.484902+0000', 'last_fullsized': '2023-12-16T21:58:40.484902+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-16T21:58:45.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2023-12-16T21:58:45.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.501035+0000', 'last_change': '2023-12-16T21:58:40.501035+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.501035+0000', 'last_undegraded': '2023-12-16T21:58:40.501035+0000', 'last_fullsized': '2023-12-16T21:58:40.501035+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2023-12-16T21:58:45.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2023-12-16T21:58:45.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.482640+0000', 'last_change': '2023-12-16T21:58:40.482640+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.482640+0000', 'last_undegraded': '2023-12-16T21:58:40.482640+0000', 'last_fullsized': '2023-12-16T21:58:40.482640+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-16T21:58:45.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1e is not active+clean 2023-12-16T21:58:45.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.529438+0000', 'last_change': '2023-12-16T21:58:39.529438+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.529438+0000', 'last_undegraded': '2023-12-16T21:58:39.529438+0000', 'last_fullsized': '2023-12-16T21:58:39.529438+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-16T21:58:45.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2023-12-16T21:58:45.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.489570+0000', 'last_change': '2023-12-16T21:58:40.489570+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.489570+0000', 'last_undegraded': '2023-12-16T21:58:40.489570+0000', 'last_fullsized': '2023-12-16T21:58:40.489570+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-16T21:58:45.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1d is not active+clean 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.478861+0000', 'last_change': '2023-12-16T21:58:39.478861+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.478861+0000', 'last_undegraded': '2023-12-16T21:58:39.478861+0000', 'last_fullsized': '2023-12-16T21:58:39.478861+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1c is not active+clean 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:39.467225+0000', 'last_change': '2023-12-16T21:58:39.467225+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.467225+0000', 'last_undegraded': '2023-12-16T21:58:39.467225+0000', 'last_fullsized': '2023-12-16T21:58:39.467225+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.493858+0000', 'last_change': '2023-12-16T21:58:40.493858+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.493858+0000', 'last_undegraded': '2023-12-16T21:58:40.493858+0000', 'last_fullsized': '2023-12-16T21:58:40.493858+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2023-12-16T21:58:45.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.494574+0000', 'last_change': '2023-12-16T21:58:40.494574+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.494574+0000', 'last_undegraded': '2023-12-16T21:58:40.494574+0000', 'last_fullsized': '2023-12-16T21:58:40.494574+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2023-12-16T21:58:45.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.483333+0000', 'last_change': '2023-12-16T21:58:40.483333+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.483333+0000', 'last_undegraded': '2023-12-16T21:58:40.483333+0000', 'last_fullsized': '2023-12-16T21:58:40.483333+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-16T21:58:45.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.19 is not active+clean 2023-12-16T21:58:45.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.478762+0000', 'last_change': '2023-12-16T21:58:39.478762+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.478762+0000', 'last_undegraded': '2023-12-16T21:58:39.478762+0000', 'last_fullsized': '2023-12-16T21:58:39.478762+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-16T21:58:45.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2023-12-16T21:58:45.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2023-12-16T21:58:45.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.491282+0000', 'last_change': '2023-12-16T21:58:40.491282+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.491282+0000', 'last_undegraded': '2023-12-16T21:58:40.491282+0000', 'last_fullsized': '2023-12-16T21:58:40.491282+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.17 is not active+clean 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.477459+0000', 'last_change': '2023-12-16T21:58:39.477459+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.477459+0000', 'last_undegraded': '2023-12-16T21:58:39.477459+0000', 'last_fullsized': '2023-12-16T21:58:39.477459+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.16 is not active+clean 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.528902+0000', 'last_change': '2023-12-16T21:58:39.528902+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.528902+0000', 'last_undegraded': '2023-12-16T21:58:39.528902+0000', 'last_fullsized': '2023-12-16T21:58:39.528902+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2023-12-16T21:58:45.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.493053+0000', 'last_change': '2023-12-16T21:58:40.493053+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.493053+0000', 'last_undegraded': '2023-12-16T21:58:40.493053+0000', 'last_fullsized': '2023-12-16T21:58:40.493053+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.14 is not active+clean 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.477394+0000', 'last_change': '2023-12-16T21:58:39.477394+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.477394+0000', 'last_undegraded': '2023-12-16T21:58:39.477394+0000', 'last_fullsized': '2023-12-16T21:58:39.477394+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.12 is not active+clean 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.482247+0000', 'last_change': '2023-12-16T21:58:39.482247+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.482247+0000', 'last_undegraded': '2023-12-16T21:58:39.482247+0000', 'last_fullsized': '2023-12-16T21:58:39.482247+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2023-12-16T21:58:45.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.496004+0000', 'last_change': '2023-12-16T21:58:40.496004+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.496004+0000', 'last_undegraded': '2023-12-16T21:58:40.496004+0000', 'last_fullsized': '2023-12-16T21:58:40.496004+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.11 is not active+clean 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.477445+0000', 'last_change': '2023-12-16T21:58:39.477445+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.477445+0000', 'last_undegraded': '2023-12-16T21:58:39.477445+0000', 'last_fullsized': '2023-12-16T21:58:39.477445+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.482543+0000', 'last_change': '2023-12-16T21:58:40.482543+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.482543+0000', 'last_undegraded': '2023-12-16T21:58:40.482543+0000', 'last_fullsized': '2023-12-16T21:58:40.482543+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.10 is not active+clean 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.478219+0000', 'last_change': '2023-12-16T21:58:39.478219+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.478219+0000', 'last_undegraded': '2023-12-16T21:58:39.478219+0000', 'last_fullsized': '2023-12-16T21:58:39.478219+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.485868+0000', 'last_change': '2023-12-16T21:58:40.485868+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.485868+0000', 'last_undegraded': '2023-12-16T21:58:40.485868+0000', 'last_fullsized': '2023-12-16T21:58:40.485868+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.f is not active+clean 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.481593+0000', 'last_change': '2023-12-16T21:58:39.481593+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.481593+0000', 'last_undegraded': '2023-12-16T21:58:39.481593+0000', 'last_fullsized': '2023-12-16T21:58:39.481593+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T21:58:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.494957+0000', 'last_change': '2023-12-16T21:58:40.494957+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.494957+0000', 'last_undegraded': '2023-12-16T21:58:40.494957+0000', 'last_fullsized': '2023-12-16T21:58:40.494957+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.b is not active+clean 2023-12-16T21:58:45.374 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.528972+0000', 'last_change': '2023-12-16T21:58:39.528972+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.528972+0000', 'last_undegraded': '2023-12-16T21:58:39.528972+0000', 'last_fullsized': '2023-12-16T21:58:39.528972+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-16T21:58:45.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2023-12-16T21:58:45.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2023-12-16T21:58:45.375 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.375 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2023-12-16T21:58:45.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.500732+0000', 'last_change': '2023-12-16T21:58:40.500732+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.500732+0000', 'last_undegraded': '2023-12-16T21:58:40.500732+0000', 'last_fullsized': '2023-12-16T21:58:40.500732+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2023-12-16T21:58:45.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.376 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.7 is not active+clean 2023-12-16T21:58:45.376 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.480909+0000', 'last_change': '2023-12-16T21:58:39.480909+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.480909+0000', 'last_undegraded': '2023-12-16T21:58:39.480909+0000', 'last_fullsized': '2023-12-16T21:58:39.480909+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.494047+0000', 'last_change': '2023-12-16T21:58:40.494047+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.494047+0000', 'last_undegraded': '2023-12-16T21:58:40.494047+0000', 'last_fullsized': '2023-12-16T21:58:40.494047+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.6 is not active+clean 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.528116+0000', 'last_change': '2023-12-16T21:58:39.528116+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.528116+0000', 'last_undegraded': '2023-12-16T21:58:39.528116+0000', 'last_fullsized': '2023-12-16T21:58:39.528116+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.4 is not active+clean 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.480710+0000', 'last_change': '2023-12-16T21:58:39.480710+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.480710+0000', 'last_undegraded': '2023-12-16T21:58:39.480710+0000', 'last_fullsized': '2023-12-16T21:58:39.480710+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T21:58:45.377 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.483887+0000', 'last_change': '2023-12-16T21:58:40.483887+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.483887+0000', 'last_undegraded': '2023-12-16T21:58:40.483887+0000', 'last_fullsized': '2023-12-16T21:58:40.483887+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3 is not active+clean 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.528182+0000', 'last_change': '2023-12-16T21:58:39.528182+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.528182+0000', 'last_undegraded': '2023-12-16T21:58:39.528182+0000', 'last_fullsized': '2023-12-16T21:58:39.528182+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2 is not active+clean 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.480584+0000', 'last_change': '2023-12-16T21:58:39.480584+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.480584+0000', 'last_undegraded': '2023-12-16T21:58:39.480584+0000', 'last_fullsized': '2023-12-16T21:58:39.480584+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-12-16T21:58:40.470714+0000', 'last_change': '2023-12-16T21:58:40.470714+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.470714+0000', 'last_undegraded': '2023-12-16T21:58:40.470714+0000', 'last_fullsized': '2023-12-16T21:58:40.470714+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2023-12-16T21:58:45.378 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.484418+0000', 'last_change': '2023-12-16T21:58:40.484418+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.484418+0000', 'last_undegraded': '2023-12-16T21:58:40.484418+0000', 'last_fullsized': '2023-12-16T21:58:40.484418+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-12-16T21:58:45.379 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.0 is not active+clean 2023-12-16T21:58:45.379 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:39.477392+0000', 'last_change': '2023-12-16T21:58:39.477392+0000', 'last_active': '2023-12-16T21:58:39.467225+0000', 'last_peered': '2023-12-16T21:58:39.467225+0000', 'last_clean': '2023-12-16T21:58:39.467225+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:39.477392+0000', 'last_undegraded': '2023-12-16T21:58:39.477392+0000', 'last_fullsized': '2023-12-16T21:58:39.477392+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:39.467225+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-12-16T21:58:45.379 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2023-12-16T21:58:45.379 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 34, 'state': 'creating+peering', 'last_fresh': '2023-12-16T21:58:40.497157+0000', 'last_change': '2023-12-16T21:58:40.497157+0000', 'last_active': '2023-12-16T21:58:40.470714+0000', 'last_peered': '2023-12-16T21:58:40.470714+0000', 'last_clean': '2023-12-16T21:58:40.470714+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-12-16T21:58:40.497157+0000', 'last_undegraded': '2023-12-16T21:58:40.497157+0000', 'last_fullsized': '2023-12-16T21:58:40.497157+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 34, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'last_clean_scrub_stamp': '2023-12-16T21:58:40.470714+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-12-16T21:58:45.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-12-16T21:58:45.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd5755ad640 1 Processor -- start 2023-12-16T21:58:45.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd5755ad640 1 -- start start 2023-12-16T21:58:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd5755ad640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd570144840 0x7fd57006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd5755ad640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd570070b70 0x7fd57006c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd5755ad640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700710e0 con 0x7fd570072180 2023-12-16T21:58:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd5755ad640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd57015f010 con 0x7fd570070b70 2023-12-16T21:58:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd5755ad640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5700713c0 con 0x7fd570144840 2023-12-16T21:58:45.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd56f7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd570070b70 0x7fd57006c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd56f7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd570070b70 0x7fd57006c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36976/0 (socket says 172.21.15.28:36976) 2023-12-16T21:58:45.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd56f7fe640 1 -- 172.21.15.28:0/2214589194 learned_addr learned my addr 172.21.15.28:0/2214589194 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:45.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.513+0000 7fd56effd640 1 --2- 172.21.15.28:0/2214589194 >> v2:172.21.15.122:3300/0 conn(0x7fd570144840 0x7fd57006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56f7fe640 1 -- 172.21.15.28:0/2214589194 >> v2:172.21.15.122:3300/0 conn(0x7fd570144840 msgr2=0x7fd57006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:45.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/2214589194 >> v2:172.21.15.122:3300/0 conn(0x7fd570144840 0x7fd57006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56f7fe640 1 -- 172.21.15.28:0/2214589194 >> v1:172.21.15.122:6789/0 conn(0x7fd570072180 legacy=0x7fd570070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:45.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56f7fe640 1 -- 172.21.15.28:0/2214589194 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700716a0 con 0x7fd570070b70 2023-12-16T21:58:45.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56effd640 1 --2- 172.21.15.28:0/2214589194 >> v2:172.21.15.122:3300/0 conn(0x7fd570144840 0x7fd57006c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:45.529 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/2214589194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570070b70 0x7fd57006c800 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd570068020 tx=0x7fd56801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cec8a2b4737a3e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.529 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56dffb640 1 -- 172.21.15.28:0/2214589194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5680aa5f0 con 0x7fd570070b70 2023-12-16T21:58:45.529 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56dffb640 1 -- 172.21.15.28:0/2214589194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5680aa750 con 0x7fd570070b70 2023-12-16T21:58:45.529 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd56dffb640 1 -- 172.21.15.28:0/2214589194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5680aaae0 con 0x7fd570070b70 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd5755ad640 1 -- 172.21.15.28:0/2214589194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570070b70 msgr2=0x7fd57006c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd5755ad640 1 --2- 172.21.15.28:0/2214589194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570070b70 0x7fd57006c800 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd570068020 tx=0x7fd56801c490 comp rx=0 tx=0).stop 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd5755ad640 1 -- 172.21.15.28:0/2214589194 shutdown_connections 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd5755ad640 1 --2- 172.21.15.28:0/2214589194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570070b70 0x7fd57006c800 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd5755ad640 1 --2- 172.21.15.28:0/2214589194 >> v2:172.21.15.122:3300/0 conn(0x7fd570144840 0x7fd57006c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd5755ad640 1 -- 172.21.15.28:0/2214589194 >> 172.21.15.28:0/2214589194 conn(0x7fd57006a530 msgr2=0x7fd57006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd5755ad640 1 -- 172.21.15.28:0/2214589194 shutdown_connections 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.517+0000 7fd5755ad640 1 -- 172.21.15.28:0/2214589194 wait complete. 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 Processor -- start 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 -- start start 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd570070b70 0x7fd57013d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd570072180 0x7fd57015aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570144840 0x7fd57013c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.530 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700b20c0 con 0x7fd570144840 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700cb630 con 0x7fd570072180 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700b2310 con 0x7fd570070b70 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570144840 0x7fd57013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570144840 0x7fd57013c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36984/0 (socket says 172.21.15.28:36984) 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 -- 172.21.15.28:0/1945525115 learned_addr learned my addr 172.21.15.28:0/1945525115 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56effd640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd570070b70 0x7fd57013d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56e7fc640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd570072180 0x7fd57015aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 -- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd570070b70 msgr2=0x7fd57013d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd570070b70 0x7fd57013d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 -- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd570072180 msgr2=0x7fd57015aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd570072180 0x7fd57015aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 -- 172.21.15.28:0/1945525115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680094d0 con 0x7fd570144840 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56e7fc640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd570072180 0x7fd57015aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:45.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd56f7fe640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570144840 0x7fd57013c380 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd56801a0c0 tx=0x7fd56801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5577fe640 1 -- 172.21.15.28:0/1945525115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5680b3800 con 0x7fd570144840 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5577fe640 1 -- 172.21.15.28:0/1945525115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5680b3960 con 0x7fd570144840 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5577fe640 1 -- 172.21.15.28:0/1945525115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5680b3d10 con 0x7fd570144840 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5700b25f0 con 0x7fd570144840 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5700b2bb0 con 0x7fd570144840 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd570069440 con 0x7fd570144840 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5577fe640 1 -- 172.21.15.28:0/1945525115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fd5680b2380 con 0x7fd570144840 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5577fe640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd55c066470 0x7fd55c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.521+0000 7fd5577fe640 1 -- 172.21.15.28:0/1945525115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5680a7070 con 0x7fd570144840 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.525+0000 7fd56effd640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd55c066470 0x7fd55c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.525+0000 7fd56effd640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd55c066470 0x7fd55c068930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd564006fd0 tx=0x7fd564008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.533 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.529+0000 7fd5577fe640 1 -- 172.21.15.28:0/1945525115 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fd5680eb990 con 0x7fd570144840 2023-12-16T21:58:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd57006d170 con 0x7fd570144840 2023-12-16T21:58:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5577fe640 1 -- 172.21.15.28:0/1945525115 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1717 (secure 0 0 0) 0x7fd5680aa5b0 con 0x7fd570144840 2023-12-16T21:58:45.746 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:45.746 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":87,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":36,"num_osds":8,"num_up_osds":8,"osd_up_since":1702763850,"num_in_osds":8,"osd_in_since":1702763841,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":51},{"state_name":"active+clean","count":24},{"state_name":"unknown","count":22}],"num_pgs":97,"num_pools":3,"num_objects":3,"data_bytes":590368,"bytes_used":246820864,"bytes_avail":772847292416,"bytes_total":773094113280,"unknown_pgs_ratio":0.22680412232875824,"inactive_pgs_ratio":0.52577316761016846,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5193}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2023-12-16T21:58:33.399590+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-12-16T21:58:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd55c066470 msgr2=0x7fd55c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd55c066470 0x7fd55c068930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd564006fd0 tx=0x7fd564008040 comp rx=0 tx=0).stop 2023-12-16T21:58:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570144840 msgr2=0x7fd57013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570144840 0x7fd57013c380 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd56801a0c0 tx=0x7fd56801acd0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 shutdown_connections 2023-12-16T21:58:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd55c066470 0x7fd55c068930 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd570144840 0x7fd57013c380 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd570072180 0x7fd57015aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 --2- 172.21.15.28:0/1945525115 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd570070b70 0x7fd57013d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.741+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 >> 172.21.15.28:0/1945525115 conn(0x7fd57006a530 msgr2=0x7fd570070460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.745+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 shutdown_connections 2023-12-16T21:58:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.745+0000 7fd5755ad640 1 -- 172.21.15.28:0/1945525115 wait complete. 2023-12-16T21:58:45.766 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-12-16T21:58:45.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:58:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80b0f12640 1 Processor -- start 2023-12-16T21:58:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80b0f12640 1 -- start start 2023-12-16T21:58:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80b0f12640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f80ac06b540 0x7f80ac06b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80b0f12640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80b0f12640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ac154a20 con 0x7f80ac154640 2023-12-16T21:58:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80b0f12640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ac06bf30 con 0x7f80ac06b540 2023-12-16T21:58:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80b0f12640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac0cc8e0 con 0x7f80ac0ce180 2023-12-16T21:58:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80aad9d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80a3fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f80ac06b540 0x7f80ac06b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80aad9d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:54386/0 (socket says 172.21.15.28:54386) 2023-12-16T21:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80aad9d640 1 -- 172.21.15.28:0/2310598591 learned_addr learned my addr 172.21.15.28:0/2310598591 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80a3fff640 1 -- 172.21.15.28:0/2310598591 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 msgr2=0x7f80ac0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80a3fff640 1 --2- 172.21.15.28:0/2310598591 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80a3fff640 1 -- 172.21.15.28:0/2310598591 >> v1:172.21.15.122:6789/0 conn(0x7f80ac154640 legacy=0x7f80ac0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80a9d9b640 1 -- 172.21.15.28:0/2310598591 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280274717 0 0) 0x7f80ac154a20 con 0x7f80ac154640 2023-12-16T21:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80a3fff640 1 -- 172.21.15.28:0/2310598591 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac0ccb70 con 0x7f80ac06b540 2023-12-16T21:58:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.889+0000 7f80aad9d640 1 --2- 172.21.15.28:0/2310598591 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80a3fff640 1 --2- 172.21.15.28:0/2310598591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac06b540 0x7f80ac06b9f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f80940097c0 tx=0x7f809401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd993a42b71df00a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80a9d9b640 1 -- 172.21.15.28:0/2310598591 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0cf2e0 con 0x7f80ac06b540 2023-12-16T21:58:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80a9d9b640 1 -- 172.21.15.28:0/2310598591 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80940ab650 con 0x7f80ac06b540 2023-12-16T21:58:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80a9d9b640 1 -- 172.21.15.28:0/2310598591 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80940ab9e0 con 0x7f80ac06b540 2023-12-16T21:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 -- 172.21.15.28:0/2310598591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac06b540 msgr2=0x7f80ac06b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 --2- 172.21.15.28:0/2310598591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac06b540 0x7f80ac06b9f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f80940097c0 tx=0x7f809401c450 comp rx=0 tx=0).stop 2023-12-16T21:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 -- 172.21.15.28:0/2310598591 shutdown_connections 2023-12-16T21:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 --2- 172.21.15.28:0/2310598591 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 --2- 172.21.15.28:0/2310598591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac06b540 0x7f80ac06b9f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 -- 172.21.15.28:0/2310598591 >> 172.21.15.28:0/2310598591 conn(0x7f80ac070d40 msgr2=0x7f80ac071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 -- 172.21.15.28:0/2310598591 shutdown_connections 2023-12-16T21:58:45.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 -- 172.21.15.28:0/2310598591 wait complete. 2023-12-16T21:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 Processor -- start 2023-12-16T21:58:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.893+0000 7f80b0f12640 1 -- start start 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80ac06b540 0x7f80ac15c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac154640 0x7f80ac151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac14c060 con 0x7f80ac06b540 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac14c340 con 0x7f80ac154640 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac14c620 con 0x7f80ac0ce180 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80ac06b540 0x7f80ac15c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80ac06b540 0x7f80ac15c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36996/0 (socket says 172.21.15.28:36996) 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 -- 172.21.15.28:0/654411625 learned_addr learned my addr 172.21.15.28:0/654411625 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80a3fff640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aad9d640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac154640 0x7f80ac151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 -- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 msgr2=0x7f80ac153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 -- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac154640 msgr2=0x7f80ac151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac154640 0x7f80ac151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 -- 172.21.15.28:0/654411625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a4009b30 con 0x7f80ac06b540 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aad9d640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac154640 0x7f80ac151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80aa59c640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80ac06b540 0x7f80ac15c170 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f809c00c3f0 tx=0x7f809c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80a2ffd640 1 -- 172.21.15.28:0/654411625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c0052d0 con 0x7f80ac06b540 2023-12-16T21:58:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80a2ffd640 1 -- 172.21.15.28:0/654411625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c005450 con 0x7f80ac06b540 2023-12-16T21:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80a2ffd640 1 -- 172.21.15.28:0/654411625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c003860 con 0x7f80ac06b540 2023-12-16T21:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8094009420 con 0x7f80ac06b540 2023-12-16T21:58:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ac14cbe0 con 0x7f80ac06b540 2023-12-16T21:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ac155430 con 0x7f80ac06b540 2023-12-16T21:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80a2ffd640 1 -- 172.21.15.28:0/654411625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f809c0039c0 con 0x7f80ac06b540 2023-12-16T21:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80a2ffd640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f808c066470 0x7f808c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80a2ffd640 1 -- 172.21.15.28:0/654411625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f809c083920 con 0x7f80ac06b540 2023-12-16T21:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.897+0000 7f80a3fff640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f808c066470 0x7f808c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.901+0000 7f80a3fff640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f808c066470 0x7f808c068930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f80940af000 tx=0x7f80940a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:45.901+0000 7f80a2ffd640 1 -- 172.21.15.28:0/654411625 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f809c050da0 con 0x7f80ac06b540 2023-12-16T21:58:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.053+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80ac0cd250 con 0x7f808c066470 2023-12-16T21:58:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.077+0000 7f80a2ffd640 1 -- 172.21.15.28:0/654411625 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280396 (secure 0 0 0) 0x7f80ac0cd250 con 0x7f808c066470 2023-12-16T21:58:46.083 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f808c066470 msgr2=0x7f808c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f808c066470 0x7f808c068930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f80940af000 tx=0x7f80940a7000 comp rx=0 tx=0).stop 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80ac06b540 msgr2=0x7f80ac15c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80ac06b540 0x7f80ac15c170 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f809c00c3f0 tx=0x7f809c00c8b0 comp rx=0 tx=0).stop 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 shutdown_connections 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f808c066470 0x7f808c068930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f80ac154640 0x7f80ac151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f80ac0ce180 0x7f80ac153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 --2- 172.21.15.28:0/654411625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80ac06b540 0x7f80ac15c170 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 >> 172.21.15.28:0/654411625 conn(0x7f80ac070d40 msgr2=0x7f80ac0af750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 shutdown_connections 2023-12-16T21:58:46.088 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:46.081+0000 7f80b0f12640 1 -- 172.21.15.28:0/654411625 wait complete. 2023-12-16T21:58:46.101 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2023-12-16T21:58:45.402766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":162,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":150,"acting":150,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241036,"kb_used_data":4052,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733684,"statfs":{"total":773094113280,"available":772847292416,"internally_reserved":0,"allocated":4149248,"data_stored":2309903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.929952"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483829+0000","last_change":"2023-12-16T21:58:40.483829+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483829+0000","last_undegraded":"2023-12-16T21:58:40.483829+0000","last_fullsized":"2023-12-16T21:58:40.483829+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.492190+0000","last_change":"2023-12-16T21:58:40.492190+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.492190+0000","last_undegraded":"2023-12-16T21:58:40.492190+0000","last_fullsized":"2023-12-16T21:58:40.492190+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481050+0000","last_change":"2023-12-16T21:58:41.516499+0000","last_active":"2023-12-16T21:58:43.481050+0000","last_peered":"2023-12-16T21:58:43.481050+0000","last_clean":"2023-12-16T21:58:43.481050+0000","last_became_active":"2023-12-16T21:58:41.516194+0000","last_became_peered":"2023-12-16T21:58:41.516194+0000","last_unstale":"2023-12-16T21:58:43.481050+0000","last_undegraded":"2023-12-16T21:58:43.481050+0000","last_fullsized":"2023-12-16T21:58:43.481050+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:25:18.864580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.499126+0000","last_change":"2023-12-16T21:58:40.499126+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.499126+0000","last_undegraded":"2023-12-16T21:58:40.499126+0000","last_fullsized":"2023-12-16T21:58:40.499126+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.505046+0000","last_change":"2023-12-16T21:58:40.505046+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.505046+0000","last_undegraded":"2023-12-16T21:58:40.505046+0000","last_fullsized":"2023-12-16T21:58:40.505046+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.488032+0000","last_change":"2023-12-16T21:58:40.488032+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.488032+0000","last_undegraded":"2023-12-16T21:58:40.488032+0000","last_fullsized":"2023-12-16T21:58:40.488032+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494042+0000","last_change":"2023-12-16T21:58:40.494042+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494042+0000","last_undegraded":"2023-12-16T21:58:40.494042+0000","last_fullsized":"2023-12-16T21:58:40.494042+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.492852+0000","last_change":"2023-12-16T21:58:40.492852+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.492852+0000","last_undegraded":"2023-12-16T21:58:40.492852+0000","last_fullsized":"2023-12-16T21:58:40.492852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480880+0000","last_change":"2023-12-16T21:58:41.528079+0000","last_active":"2023-12-16T21:58:43.480880+0000","last_peered":"2023-12-16T21:58:43.480880+0000","last_clean":"2023-12-16T21:58:43.480880+0000","last_became_active":"2023-12-16T21:58:41.527692+0000","last_became_peered":"2023-12-16T21:58:41.527692+0000","last_unstale":"2023-12-16T21:58:43.480880+0000","last_undegraded":"2023-12-16T21:58:43.480880+0000","last_fullsized":"2023-12-16T21:58:43.480880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:45:40.039820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.496638+0000","last_change":"2023-12-16T21:58:40.496638+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.496638+0000","last_undegraded":"2023-12-16T21:58:40.496638+0000","last_fullsized":"2023-12-16T21:58:40.496638+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493692+0000","last_change":"2023-12-16T21:58:40.493692+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493692+0000","last_undegraded":"2023-12-16T21:58:40.493692+0000","last_fullsized":"2023-12-16T21:58:40.493692+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.497617+0000","last_change":"2023-12-16T21:58:40.497617+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.497617+0000","last_undegraded":"2023-12-16T21:58:40.497617+0000","last_fullsized":"2023-12-16T21:58:40.497617+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481230+0000","last_change":"2023-12-16T21:58:41.563667+0000","last_active":"2023-12-16T21:58:43.481230+0000","last_peered":"2023-12-16T21:58:43.481230+0000","last_clean":"2023-12-16T21:58:43.481230+0000","last_became_active":"2023-12-16T21:58:41.563225+0000","last_became_peered":"2023-12-16T21:58:41.563225+0000","last_unstale":"2023-12-16T21:58:43.481230+0000","last_undegraded":"2023-12-16T21:58:43.481230+0000","last_fullsized":"2023-12-16T21:58:43.481230+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:43:03.724509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491447+0000","last_change":"2023-12-16T21:58:40.491447+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491447+0000","last_undegraded":"2023-12-16T21:58:40.491447+0000","last_fullsized":"2023-12-16T21:58:40.491447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.487840+0000","last_change":"2023-12-16T21:58:40.487840+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.487840+0000","last_undegraded":"2023-12-16T21:58:40.487840+0000","last_fullsized":"2023-12-16T21:58:40.487840+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.490713+0000","last_change":"2023-12-16T21:58:40.490713+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.490713+0000","last_undegraded":"2023-12-16T21:58:40.490713+0000","last_fullsized":"2023-12-16T21:58:40.490713+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491975+0000","last_change":"2023-12-16T21:58:40.491975+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491975+0000","last_undegraded":"2023-12-16T21:58:40.491975+0000","last_fullsized":"2023-12-16T21:58:40.491975+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.488011+0000","last_change":"2023-12-16T21:58:40.488011+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.488011+0000","last_undegraded":"2023-12-16T21:58:40.488011+0000","last_fullsized":"2023-12-16T21:58:40.488011+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.495037+0000","last_change":"2023-12-16T21:58:40.495037+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.495037+0000","last_undegraded":"2023-12-16T21:58:40.495037+0000","last_fullsized":"2023-12-16T21:58:40.495037+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.484902+0000","last_change":"2023-12-16T21:58:40.484902+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.484902+0000","last_undegraded":"2023-12-16T21:58:40.484902+0000","last_fullsized":"2023-12-16T21:58:40.484902+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.501035+0000","last_change":"2023-12-16T21:58:40.501035+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.501035+0000","last_undegraded":"2023-12-16T21:58:40.501035+0000","last_fullsized":"2023-12-16T21:58:40.501035+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.517957+0000","last_change":"2023-12-16T21:58:40.510463+0000","last_active":"2023-12-16T21:58:40.517957+0000","last_peered":"2023-12-16T21:58:40.517957+0000","last_clean":"2023-12-16T21:58:40.517957+0000","last_became_active":"2023-12-16T21:58:40.509883+0000","last_became_peered":"2023-12-16T21:58:40.509883+0000","last_unstale":"2023-12-16T21:58:40.517957+0000","last_undegraded":"2023-12-16T21:58:40.517957+0000","last_fullsized":"2023-12-16T21:58:40.517957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:46:52.112074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.482640+0000","last_change":"2023-12-16T21:58:40.482640+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.482640+0000","last_undegraded":"2023-12-16T21:58:40.482640+0000","last_fullsized":"2023-12-16T21:58:40.482640+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.529438+0000","last_change":"2023-12-16T21:58:39.529438+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.529438+0000","last_undegraded":"2023-12-16T21:58:39.529438+0000","last_fullsized":"2023-12-16T21:58:39.529438+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.489570+0000","last_change":"2023-12-16T21:58:40.489570+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.489570+0000","last_undegraded":"2023-12-16T21:58:40.489570+0000","last_fullsized":"2023-12-16T21:58:40.489570+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478861+0000","last_change":"2023-12-16T21:58:39.478861+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478861+0000","last_undegraded":"2023-12-16T21:58:39.478861+0000","last_fullsized":"2023-12-16T21:58:39.478861+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481374+0000","last_change":"2023-12-16T21:58:40.502942+0000","last_active":"2023-12-16T21:58:43.481374+0000","last_peered":"2023-12-16T21:58:43.481374+0000","last_clean":"2023-12-16T21:58:43.481374+0000","last_became_active":"2023-12-16T21:58:40.502586+0000","last_became_peered":"2023-12-16T21:58:40.502586+0000","last_unstale":"2023-12-16T21:58:43.481374+0000","last_undegraded":"2023-12-16T21:58:43.481374+0000","last_fullsized":"2023-12-16T21:58:43.481374+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:47:41.478553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493858+0000","last_change":"2023-12-16T21:58:40.493858+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493858+0000","last_undegraded":"2023-12-16T21:58:40.493858+0000","last_fullsized":"2023-12-16T21:58:40.493858+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522509+0000","last_change":"2023-12-16T21:58:40.516801+0000","last_active":"2023-12-16T21:58:40.522509+0000","last_peered":"2023-12-16T21:58:40.522509+0000","last_clean":"2023-12-16T21:58:40.522509+0000","last_became_active":"2023-12-16T21:58:40.515829+0000","last_became_peered":"2023-12-16T21:58:40.515829+0000","last_unstale":"2023-12-16T21:58:40.522509+0000","last_undegraded":"2023-12-16T21:58:40.522509+0000","last_fullsized":"2023-12-16T21:58:40.522509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:26:05.935472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494574+0000","last_change":"2023-12-16T21:58:40.494574+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494574+0000","last_undegraded":"2023-12-16T21:58:40.494574+0000","last_fullsized":"2023-12-16T21:58:40.494574+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514237+0000","last_change":"2023-12-16T21:58:40.501591+0000","last_active":"2023-12-16T21:58:40.514237+0000","last_peered":"2023-12-16T21:58:40.514237+0000","last_clean":"2023-12-16T21:58:40.514237+0000","last_became_active":"2023-12-16T21:58:40.500870+0000","last_became_peered":"2023-12-16T21:58:40.500870+0000","last_unstale":"2023-12-16T21:58:40.514237+0000","last_undegraded":"2023-12-16T21:58:40.514237+0000","last_fullsized":"2023-12-16T21:58:40.514237+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:34:30.857548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483333+0000","last_change":"2023-12-16T21:58:40.483333+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483333+0000","last_undegraded":"2023-12-16T21:58:40.483333+0000","last_fullsized":"2023-12-16T21:58:40.483333+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478762+0000","last_change":"2023-12-16T21:58:39.478762+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478762+0000","last_undegraded":"2023-12-16T21:58:39.478762+0000","last_fullsized":"2023-12-16T21:58:39.478762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519509+0000","last_change":"2023-12-16T21:58:40.511738+0000","last_active":"2023-12-16T21:58:40.519509+0000","last_peered":"2023-12-16T21:58:40.519509+0000","last_clean":"2023-12-16T21:58:40.519509+0000","last_became_active":"2023-12-16T21:58:40.510752+0000","last_became_peered":"2023-12-16T21:58:40.510752+0000","last_unstale":"2023-12-16T21:58:40.519509+0000","last_undegraded":"2023-12-16T21:58:40.519509+0000","last_fullsized":"2023-12-16T21:58:40.519509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:47:01.519007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.491282+0000","last_change":"2023-12-16T21:58:40.491282+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.491282+0000","last_undegraded":"2023-12-16T21:58:40.491282+0000","last_fullsized":"2023-12-16T21:58:40.491282+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477459+0000","last_change":"2023-12-16T21:58:39.477459+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477459+0000","last_undegraded":"2023-12-16T21:58:39.477459+0000","last_fullsized":"2023-12-16T21:58:39.477459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528902+0000","last_change":"2023-12-16T21:58:39.528902+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528902+0000","last_undegraded":"2023-12-16T21:58:39.528902+0000","last_fullsized":"2023-12-16T21:58:39.528902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.523268+0000","last_change":"2023-12-16T21:58:40.518810+0000","last_active":"2023-12-16T21:58:40.523268+0000","last_peered":"2023-12-16T21:58:40.523268+0000","last_clean":"2023-12-16T21:58:40.523268+0000","last_became_active":"2023-12-16T21:58:40.518226+0000","last_became_peered":"2023-12-16T21:58:40.518226+0000","last_unstale":"2023-12-16T21:58:40.523268+0000","last_undegraded":"2023-12-16T21:58:40.523268+0000","last_fullsized":"2023-12-16T21:58:40.523268+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:37:11.035240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.493053+0000","last_change":"2023-12-16T21:58:40.493053+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.493053+0000","last_undegraded":"2023-12-16T21:58:40.493053+0000","last_fullsized":"2023-12-16T21:58:40.493053+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477394+0000","last_change":"2023-12-16T21:58:39.477394+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477394+0000","last_undegraded":"2023-12-16T21:58:39.477394+0000","last_fullsized":"2023-12-16T21:58:39.477394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519501+0000","last_change":"2023-12-16T21:58:40.511320+0000","last_active":"2023-12-16T21:58:40.519501+0000","last_peered":"2023-12-16T21:58:40.519501+0000","last_clean":"2023-12-16T21:58:40.519501+0000","last_became_active":"2023-12-16T21:58:40.510584+0000","last_became_peered":"2023-12-16T21:58:40.510584+0000","last_unstale":"2023-12-16T21:58:40.519501+0000","last_undegraded":"2023-12-16T21:58:40.519501+0000","last_fullsized":"2023-12-16T21:58:40.519501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:49:36.075932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.482247+0000","last_change":"2023-12-16T21:58:39.482247+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.482247+0000","last_undegraded":"2023-12-16T21:58:39.482247+0000","last_fullsized":"2023-12-16T21:58:39.482247+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.496004+0000","last_change":"2023-12-16T21:58:40.496004+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.496004+0000","last_undegraded":"2023-12-16T21:58:40.496004+0000","last_fullsized":"2023-12-16T21:58:40.496004+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477445+0000","last_change":"2023-12-16T21:58:39.477445+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477445+0000","last_undegraded":"2023-12-16T21:58:39.477445+0000","last_fullsized":"2023-12-16T21:58:39.477445+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.482543+0000","last_change":"2023-12-16T21:58:40.482543+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.482543+0000","last_undegraded":"2023-12-16T21:58:40.482543+0000","last_fullsized":"2023-12-16T21:58:40.482543+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.478219+0000","last_change":"2023-12-16T21:58:39.478219+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.478219+0000","last_undegraded":"2023-12-16T21:58:39.478219+0000","last_fullsized":"2023-12-16T21:58:39.478219+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.485868+0000","last_change":"2023-12-16T21:58:40.485868+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.485868+0000","last_undegraded":"2023-12-16T21:58:40.485868+0000","last_fullsized":"2023-12-16T21:58:40.485868+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.481593+0000","last_change":"2023-12-16T21:58:39.481593+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.481593+0000","last_undegraded":"2023-12-16T21:58:39.481593+0000","last_fullsized":"2023-12-16T21:58:39.481593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494957+0000","last_change":"2023-12-16T21:58:40.494957+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494957+0000","last_undegraded":"2023-12-16T21:58:40.494957+0000","last_fullsized":"2023-12-16T21:58:40.494957+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522371+0000","last_change":"2023-12-16T21:58:40.515564+0000","last_active":"2023-12-16T21:58:40.522371+0000","last_peered":"2023-12-16T21:58:40.522371+0000","last_clean":"2023-12-16T21:58:40.522371+0000","last_became_active":"2023-12-16T21:58:40.514839+0000","last_became_peered":"2023-12-16T21:58:40.514839+0000","last_unstale":"2023-12-16T21:58:40.522371+0000","last_undegraded":"2023-12-16T21:58:40.522371+0000","last_fullsized":"2023-12-16T21:58:40.522371+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:58:46.240719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514447+0000","last_change":"2023-12-16T21:58:40.499987+0000","last_active":"2023-12-16T21:58:40.514447+0000","last_peered":"2023-12-16T21:58:40.514447+0000","last_clean":"2023-12-16T21:58:40.514447+0000","last_became_active":"2023-12-16T21:58:40.499198+0000","last_became_peered":"2023-12-16T21:58:40.499198+0000","last_unstale":"2023-12-16T21:58:40.514447+0000","last_undegraded":"2023-12-16T21:58:40.514447+0000","last_fullsized":"2023-12-16T21:58:40.514447+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:53.492867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481468+0000","last_change":"2023-12-16T21:58:41.544683+0000","last_active":"2023-12-16T21:58:43.481468+0000","last_peered":"2023-12-16T21:58:43.481468+0000","last_clean":"2023-12-16T21:58:43.481468+0000","last_became_active":"2023-12-16T21:58:41.544362+0000","last_became_peered":"2023-12-16T21:58:41.544362+0000","last_unstale":"2023-12-16T21:58:43.481468+0000","last_undegraded":"2023-12-16T21:58:43.481468+0000","last_fullsized":"2023-12-16T21:58:43.481468+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:42:05.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514335+0000","last_change":"2023-12-16T21:58:40.503566+0000","last_active":"2023-12-16T21:58:40.514335+0000","last_peered":"2023-12-16T21:58:40.514335+0000","last_clean":"2023-12-16T21:58:40.514335+0000","last_became_active":"2023-12-16T21:58:40.502798+0000","last_became_peered":"2023-12-16T21:58:40.502798+0000","last_unstale":"2023-12-16T21:58:40.514335+0000","last_undegraded":"2023-12-16T21:58:40.514335+0000","last_fullsized":"2023-12-16T21:58:40.514335+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:06:31.466515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480781+0000","last_change":"2023-12-16T21:58:41.567440+0000","last_active":"2023-12-16T21:58:43.480781+0000","last_peered":"2023-12-16T21:58:43.480781+0000","last_clean":"2023-12-16T21:58:43.480781+0000","last_became_active":"2023-12-16T21:58:41.567135+0000","last_became_peered":"2023-12-16T21:58:41.567135+0000","last_unstale":"2023-12-16T21:58:43.480781+0000","last_undegraded":"2023-12-16T21:58:43.480781+0000","last_fullsized":"2023-12-16T21:58:43.480781+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:37:41.637920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528972+0000","last_change":"2023-12-16T21:58:39.528972+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528972+0000","last_undegraded":"2023-12-16T21:58:39.528972+0000","last_fullsized":"2023-12-16T21:58:39.528972+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481262+0000","last_change":"2023-12-16T21:58:41.532084+0000","last_active":"2023-12-16T21:58:43.481262+0000","last_peered":"2023-12-16T21:58:43.481262+0000","last_clean":"2023-12-16T21:58:43.481262+0000","last_became_active":"2023-12-16T21:58:41.529955+0000","last_became_peered":"2023-12-16T21:58:41.529955+0000","last_unstale":"2023-12-16T21:58:43.481262+0000","last_undegraded":"2023-12-16T21:58:43.481262+0000","last_fullsized":"2023-12-16T21:58:43.481262+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:07:34.643469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.512812+0000","last_change":"2023-12-16T21:58:40.502158+0000","last_active":"2023-12-16T21:58:40.512812+0000","last_peered":"2023-12-16T21:58:40.512812+0000","last_clean":"2023-12-16T21:58:40.512812+0000","last_became_active":"2023-12-16T21:58:40.501224+0000","last_became_peered":"2023-12-16T21:58:40.501224+0000","last_unstale":"2023-12-16T21:58:40.512812+0000","last_undegraded":"2023-12-16T21:58:40.512812+0000","last_fullsized":"2023-12-16T21:58:40.512812+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:11:13.641738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480758+0000","last_change":"2023-12-16T21:58:41.520385+0000","last_active":"2023-12-16T21:58:43.480758+0000","last_peered":"2023-12-16T21:58:43.480758+0000","last_clean":"2023-12-16T21:58:43.480758+0000","last_became_active":"2023-12-16T21:58:41.520005+0000","last_became_peered":"2023-12-16T21:58:41.520005+0000","last_unstale":"2023-12-16T21:58:43.480758+0000","last_undegraded":"2023-12-16T21:58:43.480758+0000","last_fullsized":"2023-12-16T21:58:43.480758+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:34:53.394229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.514394+0000","last_change":"2023-12-16T21:58:40.503347+0000","last_active":"2023-12-16T21:58:40.514394+0000","last_peered":"2023-12-16T21:58:40.514394+0000","last_clean":"2023-12-16T21:58:40.514394+0000","last_became_active":"2023-12-16T21:58:40.502397+0000","last_became_peered":"2023-12-16T21:58:40.502397+0000","last_unstale":"2023-12-16T21:58:40.514394+0000","last_undegraded":"2023-12-16T21:58:40.514394+0000","last_fullsized":"2023-12-16T21:58:40.514394+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:12:26.461737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.500732+0000","last_change":"2023-12-16T21:58:40.500732+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.500732+0000","last_undegraded":"2023-12-16T21:58:40.500732+0000","last_fullsized":"2023-12-16T21:58:40.500732+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.523302+0000","last_change":"2023-12-16T21:58:40.519887+0000","last_active":"2023-12-16T21:58:40.523302+0000","last_peered":"2023-12-16T21:58:40.523302+0000","last_clean":"2023-12-16T21:58:40.523302+0000","last_became_active":"2023-12-16T21:58:40.518760+0000","last_became_peered":"2023-12-16T21:58:40.518760+0000","last_unstale":"2023-12-16T21:58:40.523302+0000","last_undegraded":"2023-12-16T21:58:40.523302+0000","last_fullsized":"2023-12-16T21:58:40.523302+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:18:57.185826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481217+0000","last_change":"2023-12-16T21:58:41.541134+0000","last_active":"2023-12-16T21:58:43.481217+0000","last_peered":"2023-12-16T21:58:43.481217+0000","last_clean":"2023-12-16T21:58:43.481217+0000","last_became_active":"2023-12-16T21:58:41.540881+0000","last_became_peered":"2023-12-16T21:58:41.540881+0000","last_unstale":"2023-12-16T21:58:43.481217+0000","last_undegraded":"2023-12-16T21:58:43.481217+0000","last_fullsized":"2023-12-16T21:58:43.481217+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:30:06.360464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480909+0000","last_change":"2023-12-16T21:58:39.480909+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480909+0000","last_undegraded":"2023-12-16T21:58:39.480909+0000","last_fullsized":"2023-12-16T21:58:39.480909+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.494047+0000","last_change":"2023-12-16T21:58:40.494047+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.494047+0000","last_undegraded":"2023-12-16T21:58:40.494047+0000","last_fullsized":"2023-12-16T21:58:40.494047+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528116+0000","last_change":"2023-12-16T21:58:39.528116+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528116+0000","last_undegraded":"2023-12-16T21:58:39.528116+0000","last_fullsized":"2023-12-16T21:58:39.528116+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480710+0000","last_change":"2023-12-16T21:58:39.480710+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480710+0000","last_undegraded":"2023-12-16T21:58:39.480710+0000","last_fullsized":"2023-12-16T21:58:39.480710+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.483887+0000","last_change":"2023-12-16T21:58:40.483887+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.483887+0000","last_undegraded":"2023-12-16T21:58:40.483887+0000","last_fullsized":"2023-12-16T21:58:40.483887+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.528182+0000","last_change":"2023-12-16T21:58:39.528182+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.528182+0000","last_undegraded":"2023-12-16T21:58:39.528182+0000","last_fullsized":"2023-12-16T21:58:39.528182+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.480584+0000","last_change":"2023-12-16T21:58:39.480584+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.480584+0000","last_undegraded":"2023-12-16T21:58:39.480584+0000","last_fullsized":"2023-12-16T21:58:39.480584+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:58:40.470714+0000","last_change":"2023-12-16T21:58:40.470714+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.470714+0000","last_undegraded":"2023-12-16T21:58:40.470714+0000","last_fullsized":"2023-12-16T21:58:40.470714+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.522536+0000","last_change":"2023-12-16T21:58:40.516094+0000","last_active":"2023-12-16T21:58:40.522536+0000","last_peered":"2023-12-16T21:58:40.522536+0000","last_clean":"2023-12-16T21:58:40.522536+0000","last_became_active":"2023-12-16T21:58:40.515413+0000","last_became_peered":"2023-12-16T21:58:40.515413+0000","last_unstale":"2023-12-16T21:58:40.522536+0000","last_undegraded":"2023-12-16T21:58:40.522536+0000","last_fullsized":"2023-12-16T21:58:40.522536+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:48:35.936102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.484418+0000","last_change":"2023-12-16T21:58:40.484418+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.484418+0000","last_undegraded":"2023-12-16T21:58:40.484418+0000","last_fullsized":"2023-12-16T21:58:40.484418+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2023-12-16T21:58:39.477392+0000","last_change":"2023-12-16T21:58:39.477392+0000","last_active":"2023-12-16T21:58:39.467225+0000","last_peered":"2023-12-16T21:58:39.467225+0000","last_clean":"2023-12-16T21:58:39.467225+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:39.477392+0000","last_undegraded":"2023-12-16T21:58:39.477392+0000","last_fullsized":"2023-12-16T21:58:39.477392+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":34,"state":"creating+peering","last_fresh":"2023-12-16T21:58:40.497157+0000","last_change":"2023-12-16T21:58:40.497157+0000","last_active":"2023-12-16T21:58:40.470714+0000","last_peered":"2023-12-16T21:58:40.470714+0000","last_clean":"2023-12-16T21:58:40.470714+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:58:40.497157+0000","last_undegraded":"2023-12-16T21:58:40.497157+0000","last_fullsized":"2023-12-16T21:58:40.497157+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:58:40.519500+0000","last_change":"2023-12-16T21:58:40.516743+0000","last_active":"2023-12-16T21:58:40.519500+0000","last_peered":"2023-12-16T21:58:40.519500+0000","last_clean":"2023-12-16T21:58:40.519500+0000","last_became_active":"2023-12-16T21:58:40.515578+0000","last_became_peered":"2023-12-16T21:58:40.515578+0000","last_unstale":"2023-12-16T21:58:40.519500+0000","last_undegraded":"2023-12-16T21:58:40.519500+0000","last_fullsized":"2023-12-16T21:58:40.519500+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:18:32.213979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":172,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:58:39.484154+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:58:39.484154+0000","last_peered":"2023-12-16T21:58:39.484154+0000","last_clean":"2023-12-16T21:58:39.484154+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:58:39.484154+0000","last_undegraded":"2023-12-16T21:58:39.484154+0000","last_fullsized":"2023-12-16T21:58:39.484154+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":5},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476754,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":732948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":1.1779999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":1.26}]},{"osd":4,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.81599999999999995}]},{"osd":5,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.0620000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.109}]}]},{"osd":6,"up_from":16,"seq":68719476753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.66800000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.71899999999999997}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.754}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":1.083}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":5.5620000000000003,"5min":5.5620000000000003,"15min":5.5620000000000003},"last":1.0549999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":0.93200000000000005}]}]},{"osd":5,"up_from":16,"seq":68719476754,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.56200000000000006}]},{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.72999999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.0569999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.98399999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.018}]},{"osd":6,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.254,"5min":3.254,"15min":3.254},"last":0.95199999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.512,"5min":2.512,"15min":2.512},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":1.339}]}]},{"osd":4,"up_from":16,"seq":68719476752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":372736,"data_stored":142580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.79100000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.91600000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.998},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.95599999999999996}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.123}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.208},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.173}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":1.085}]}]},{"osd":3,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.76700000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.80700000000000005}]}]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":360448,"data_stored":136313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.371},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.435},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.64300000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.65900000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.87}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.70599999999999996}]}]},{"osd":0,"up_from":15,"seq":64424509457,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":942080,"data_stored":720864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.79700000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.85599999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":1.21},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.046}]},{"osd":5,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":1.327},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.2569999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.081,"5min":2.081,"15min":2.081},"last":1.1719999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.3779999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":155725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.71099999999999997}]},{"osd":5,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":0.81200000000000006}]},{"osd":6,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.94999999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.86499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:58:46.119 INFO:tasks.ceph.mds.a.smithi028.stderr:2023-12-16T21:58:46.113+0000 7fbc343ec640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:58:49.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:58:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b941db640 1 Processor -- start 2023-12-16T21:58:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b941db640 1 -- start start 2023-12-16T21:58:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b941db640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b8c06c9d0 0x7f4b8c070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b941db640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c072180 0x7f4b8c0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b941db640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8c0710e0 con 0x7f4b8c144840 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b941db640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c15f010 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b941db640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c070ee0 con 0x7f4b8c072180 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b91776640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b8c06c9d0 0x7f4b8c070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b91776640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b8c06c9d0 0x7f4b8c070460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36152/0 (socket says 172.21.15.28:36152) 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b91776640 1 -- 172.21.15.28:0/884356602 learned_addr learned my addr 172.21.15.28:0/884356602 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b92778640 1 --2- 172.21.15.28:0/884356602 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c072180 0x7f4b8c0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b91776640 1 -- 172.21.15.28:0/884356602 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c072180 msgr2=0x7f4b8c0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b91776640 1 --2- 172.21.15.28:0/884356602 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c072180 0x7f4b8c0709a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b91776640 1 -- 172.21.15.28:0/884356602 >> v1:172.21.15.122:6789/0 conn(0x7f4b8c144840 legacy=0x7f4b8c06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.237+0000 7f4b91776640 1 -- 172.21.15.28:0/884356602 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c0713c0 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91776640 1 --2- 172.21.15.28:0/884356602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c070460 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009870 tx=0x7f4b7c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713e3f6c96749744 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b90f75640 1 -- 172.21.15.28:0/884356602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0aa670 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b90f75640 1 -- 172.21.15.28:0/884356602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c0aa7d0 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b90f75640 1 -- 172.21.15.28:0/884356602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0aab60 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- 172.21.15.28:0/884356602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 msgr2=0x7f4b8c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 --2- 172.21.15.28:0/884356602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c070460 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009870 tx=0x7f4b7c01c510 comp rx=0 tx=0).stop 2023-12-16T21:58:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- 172.21.15.28:0/884356602 shutdown_connections 2023-12-16T21:58:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 --2- 172.21.15.28:0/884356602 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c072180 0x7f4b8c0709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 --2- 172.21.15.28:0/884356602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c070460 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- 172.21.15.28:0/884356602 >> 172.21.15.28:0/884356602 conn(0x7f4b8c06a530 msgr2=0x7f4b8c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- 172.21.15.28:0/884356602 shutdown_connections 2023-12-16T21:58:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- 172.21.15.28:0/884356602 wait complete. 2023-12-16T21:58:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 Processor -- start 2023-12-16T21:58:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- start start 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b8c072180 0x7f4b8c142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c144840 0x7f4b8c142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1f5b80 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1f4f70 con 0x7f4b8c072180 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1f5250 con 0x7f4b8c144840 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36162/0 (socket says 172.21.15.28:36162) 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 -- 172.21.15.28:0/4080814895 learned_addr learned my addr 172.21.15.28:0/4080814895 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91776640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b8c072180 0x7f4b8c142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b92778640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c144840 0x7f4b8c142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 -- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c144840 msgr2=0x7f4b8c142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c144840 0x7f4b8c142cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 -- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b8c072180 msgr2=0x7f4b8c142780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b8c072180 0x7f4b8c142780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 -- 172.21.15.28:0/4080814895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0094d0 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b91f77640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c159c80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c3c0 tx=0x7f4b8800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b827fc640 1 -- 172.21.15.28:0/4080814895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b880052d0 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b827fc640 1 -- 172.21.15.28:0/4080814895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b88005430 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.241+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c1f5530 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.245+0000 7f4b827fc640 1 -- 172.21.15.28:0/4080814895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88003840 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.245+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c13c530 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.245+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c069440 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.245+0000 7f4b827fc640 1 -- 172.21.15.28:0/4080814895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4b880039a0 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.245+0000 7f4b827fc640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.245+0000 7f4b827fc640 1 -- 172.21.15.28:0/4080814895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b88083320 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.245+0000 7f4b91776640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.245+0000 7f4b91776640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0ae000 tx=0x7f4b7c0a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.249+0000 7f4b827fc640 1 -- 172.21.15.28:0/4080814895 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4b88050830 con 0x7f4b8c06c9d0 2023-12-16T21:58:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.401+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b8c144c20 con 0x7f4b70066470 2023-12-16T21:58:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.425+0000 7f4b827fc640 1 -- 172.21.15.28:0/4080814895 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287711 (secure 0 0 0) 0x7f4b7c0aa940 con 0x7f4b70066470 2023-12-16T21:58:49.431 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-16T21:58:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 msgr2=0x7f4b70068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0ae000 tx=0x7f4b7c0a7000 comp rx=0 tx=0).stop 2023-12-16T21:58:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 msgr2=0x7f4b8c159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c159c80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c3c0 tx=0x7f4b8800c880 comp rx=0 tx=0).stop 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 shutdown_connections 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4b70066470 0x7f4b70068930 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4b8c144840 0x7f4b8c142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b8c072180 0x7f4b8c142780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 --2- 172.21.15.28:0/4080814895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b8c06c9d0 0x7f4b8c159c80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 >> 172.21.15.28:0/4080814895 conn(0x7f4b8c06a530 msgr2=0x7f4b8c15c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 shutdown_connections 2023-12-16T21:58:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.429+0000 7f4b941db640 1 -- 172.21.15.28:0/4080814895 wait complete. 2023-12-16T21:58:49.452 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2023-12-16T21:58:49.404310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241316,"kb_used_data":4328,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754733404,"statfs":{"total":773094113280,"available":772847005696,"internally_reserved":0,"allocated":4431872,"data_stored":2430796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":88,"apply_latency_ms":88,"commit_latency_ns":88000000,"apply_latency_ns":88000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002747"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.510329+0000","last_change":"2023-12-16T21:58:41.555571+0000","last_active":"2023-12-16T21:58:42.510329+0000","last_peered":"2023-12-16T21:58:42.510329+0000","last_clean":"2023-12-16T21:58:42.510329+0000","last_became_active":"2023-12-16T21:58:41.554847+0000","last_became_peered":"2023-12-16T21:58:41.554847+0000","last_unstale":"2023-12-16T21:58:42.510329+0000","last_undegraded":"2023-12-16T21:58:42.510329+0000","last_fullsized":"2023-12-16T21:58:42.510329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:35:24.981673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.045650+0000","last_change":"2023-12-16T21:58:41.520660+0000","last_active":"2023-12-16T21:58:43.045650+0000","last_peered":"2023-12-16T21:58:43.045650+0000","last_clean":"2023-12-16T21:58:43.045650+0000","last_became_active":"2023-12-16T21:58:41.520379+0000","last_became_peered":"2023-12-16T21:58:41.520379+0000","last_unstale":"2023-12-16T21:58:43.045650+0000","last_undegraded":"2023-12-16T21:58:43.045650+0000","last_fullsized":"2023-12-16T21:58:43.045650+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:53:20.775769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.508813+0000","last_change":"2023-12-16T21:58:41.553088+0000","last_active":"2023-12-16T21:58:42.508813+0000","last_peered":"2023-12-16T21:58:42.508813+0000","last_clean":"2023-12-16T21:58:42.508813+0000","last_became_active":"2023-12-16T21:58:41.552427+0000","last_became_peered":"2023-12-16T21:58:41.552427+0000","last_unstale":"2023-12-16T21:58:42.508813+0000","last_undegraded":"2023-12-16T21:58:42.508813+0000","last_fullsized":"2023-12-16T21:58:42.508813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:52:50.902985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.505856+0000","last_change":"2023-12-16T21:58:41.567451+0000","last_active":"2023-12-16T21:58:42.505856+0000","last_peered":"2023-12-16T21:58:42.505856+0000","last_clean":"2023-12-16T21:58:42.505856+0000","last_became_active":"2023-12-16T21:58:41.567107+0000","last_became_peered":"2023-12-16T21:58:41.567107+0000","last_unstale":"2023-12-16T21:58:42.505856+0000","last_undegraded":"2023-12-16T21:58:42.505856+0000","last_fullsized":"2023-12-16T21:58:42.505856+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:26:56.270290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481050+0000","last_change":"2023-12-16T21:58:41.516499+0000","last_active":"2023-12-16T21:58:43.481050+0000","last_peered":"2023-12-16T21:58:43.481050+0000","last_clean":"2023-12-16T21:58:43.481050+0000","last_became_active":"2023-12-16T21:58:41.516194+0000","last_became_peered":"2023-12-16T21:58:41.516194+0000","last_unstale":"2023-12-16T21:58:43.481050+0000","last_undegraded":"2023-12-16T21:58:43.481050+0000","last_fullsized":"2023-12-16T21:58:43.481050+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:25:18.864580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.044900+0000","last_change":"2023-12-16T21:58:41.554376+0000","last_active":"2023-12-16T21:58:43.044900+0000","last_peered":"2023-12-16T21:58:43.044900+0000","last_clean":"2023-12-16T21:58:43.044900+0000","last_became_active":"2023-12-16T21:58:41.554023+0000","last_became_peered":"2023-12-16T21:58:41.554023+0000","last_unstale":"2023-12-16T21:58:43.044900+0000","last_undegraded":"2023-12-16T21:58:43.044900+0000","last_fullsized":"2023-12-16T21:58:43.044900+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:52:50.902985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480863+0000","last_change":"2023-12-16T21:58:41.536837+0000","last_active":"2023-12-16T21:58:43.480863+0000","last_peered":"2023-12-16T21:58:43.480863+0000","last_clean":"2023-12-16T21:58:43.480863+0000","last_became_active":"2023-12-16T21:58:41.536066+0000","last_became_peered":"2023-12-16T21:58:41.536066+0000","last_unstale":"2023-12-16T21:58:43.480863+0000","last_undegraded":"2023-12-16T21:58:43.480863+0000","last_fullsized":"2023-12-16T21:58:43.480863+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:39.342482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480475+0000","last_change":"2023-12-16T21:58:41.536820+0000","last_active":"2023-12-16T21:58:43.480475+0000","last_peered":"2023-12-16T21:58:43.480475+0000","last_clean":"2023-12-16T21:58:43.480475+0000","last_became_active":"2023-12-16T21:58:41.536024+0000","last_became_peered":"2023-12-16T21:58:41.536024+0000","last_unstale":"2023-12-16T21:58:43.480475+0000","last_undegraded":"2023-12-16T21:58:43.480475+0000","last_fullsized":"2023-12-16T21:58:43.480475+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:52:50.902985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.482986+0000","last_change":"2023-12-16T21:58:41.567397+0000","last_active":"2023-12-16T21:58:43.482986+0000","last_peered":"2023-12-16T21:58:43.482986+0000","last_clean":"2023-12-16T21:58:43.482986+0000","last_became_active":"2023-12-16T21:58:41.566819+0000","last_became_peered":"2023-12-16T21:58:41.566819+0000","last_unstale":"2023-12-16T21:58:43.482986+0000","last_undegraded":"2023-12-16T21:58:43.482986+0000","last_fullsized":"2023-12-16T21:58:43.482986+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:38:28.128229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.506543+0000","last_change":"2023-12-16T21:58:41.536514+0000","last_active":"2023-12-16T21:58:42.506543+0000","last_peered":"2023-12-16T21:58:42.506543+0000","last_clean":"2023-12-16T21:58:42.506543+0000","last_became_active":"2023-12-16T21:58:41.536079+0000","last_became_peered":"2023-12-16T21:58:41.536079+0000","last_unstale":"2023-12-16T21:58:42.506543+0000","last_undegraded":"2023-12-16T21:58:42.506543+0000","last_fullsized":"2023-12-16T21:58:42.506543+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:38:28.128229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481863+0000","last_change":"2023-12-16T21:58:41.543098+0000","last_active":"2023-12-16T21:58:43.481863+0000","last_peered":"2023-12-16T21:58:43.481863+0000","last_clean":"2023-12-16T21:58:43.481863+0000","last_became_active":"2023-12-16T21:58:41.537770+0000","last_became_peered":"2023-12-16T21:58:41.537770+0000","last_unstale":"2023-12-16T21:58:43.481863+0000","last_undegraded":"2023-12-16T21:58:43.481863+0000","last_fullsized":"2023-12-16T21:58:43.481863+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:02:26.903073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.510597+0000","last_change":"2023-12-16T21:58:41.548265+0000","last_active":"2023-12-16T21:58:42.510597+0000","last_peered":"2023-12-16T21:58:42.510597+0000","last_clean":"2023-12-16T21:58:42.510597+0000","last_became_active":"2023-12-16T21:58:41.541710+0000","last_became_peered":"2023-12-16T21:58:41.541710+0000","last_unstale":"2023-12-16T21:58:42.510597+0000","last_undegraded":"2023-12-16T21:58:42.510597+0000","last_fullsized":"2023-12-16T21:58:42.510597+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:26:27.870031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480880+0000","last_change":"2023-12-16T21:58:41.528079+0000","last_active":"2023-12-16T21:58:43.480880+0000","last_peered":"2023-12-16T21:58:43.480880+0000","last_clean":"2023-12-16T21:58:43.480880+0000","last_became_active":"2023-12-16T21:58:41.527692+0000","last_became_peered":"2023-12-16T21:58:41.527692+0000","last_unstale":"2023-12-16T21:58:43.480880+0000","last_undegraded":"2023-12-16T21:58:43.480880+0000","last_fullsized":"2023-12-16T21:58:43.480880+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:45:40.039820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.045048+0000","last_change":"2023-12-16T21:58:41.517353+0000","last_active":"2023-12-16T21:58:43.045048+0000","last_peered":"2023-12-16T21:58:43.045048+0000","last_clean":"2023-12-16T21:58:43.045048+0000","last_became_active":"2023-12-16T21:58:41.516972+0000","last_became_peered":"2023-12-16T21:58:41.516972+0000","last_unstale":"2023-12-16T21:58:43.045048+0000","last_undegraded":"2023-12-16T21:58:43.045048+0000","last_fullsized":"2023-12-16T21:58:43.045048+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:13:46.595747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.508785+0000","last_change":"2023-12-16T21:58:41.562573+0000","last_active":"2023-12-16T21:58:42.508785+0000","last_peered":"2023-12-16T21:58:42.508785+0000","last_clean":"2023-12-16T21:58:42.508785+0000","last_became_active":"2023-12-16T21:58:41.555176+0000","last_became_peered":"2023-12-16T21:58:41.555176+0000","last_unstale":"2023-12-16T21:58:42.508785+0000","last_undegraded":"2023-12-16T21:58:42.508785+0000","last_fullsized":"2023-12-16T21:58:42.508785+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:59:35.025704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481583+0000","last_change":"2023-12-16T21:58:41.544386+0000","last_active":"2023-12-16T21:58:43.481583+0000","last_peered":"2023-12-16T21:58:43.481583+0000","last_clean":"2023-12-16T21:58:43.481583+0000","last_became_active":"2023-12-16T21:58:41.542725+0000","last_became_peered":"2023-12-16T21:58:41.542725+0000","last_unstale":"2023-12-16T21:58:43.481583+0000","last_undegraded":"2023-12-16T21:58:43.481583+0000","last_fullsized":"2023-12-16T21:58:43.481583+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:36:34.677954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.482044+0000","last_change":"2023-12-16T21:58:41.536923+0000","last_active":"2023-12-16T21:58:43.482044+0000","last_peered":"2023-12-16T21:58:43.482044+0000","last_clean":"2023-12-16T21:58:43.482044+0000","last_became_active":"2023-12-16T21:58:41.536176+0000","last_became_peered":"2023-12-16T21:58:41.536176+0000","last_unstale":"2023-12-16T21:58:43.482044+0000","last_undegraded":"2023-12-16T21:58:43.482044+0000","last_fullsized":"2023-12-16T21:58:43.482044+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:21:37.626009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.045791+0000","last_change":"2023-12-16T21:58:41.533264+0000","last_active":"2023-12-16T21:58:43.045791+0000","last_peered":"2023-12-16T21:58:43.045791+0000","last_clean":"2023-12-16T21:58:43.045791+0000","last_became_active":"2023-12-16T21:58:41.527193+0000","last_became_peered":"2023-12-16T21:58:41.527193+0000","last_unstale":"2023-12-16T21:58:43.045791+0000","last_undegraded":"2023-12-16T21:58:43.045791+0000","last_fullsized":"2023-12-16T21:58:43.045791+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:26:27.870031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.045202+0000","last_change":"2023-12-16T21:58:41.534146+0000","last_active":"2023-12-16T21:58:43.045202+0000","last_peered":"2023-12-16T21:58:43.045202+0000","last_clean":"2023-12-16T21:58:43.045202+0000","last_became_active":"2023-12-16T21:58:41.526210+0000","last_became_peered":"2023-12-16T21:58:41.526210+0000","last_unstale":"2023-12-16T21:58:43.045202+0000","last_undegraded":"2023-12-16T21:58:43.045202+0000","last_fullsized":"2023-12-16T21:58:43.045202+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:40:46.601255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481230+0000","last_change":"2023-12-16T21:58:41.563667+0000","last_active":"2023-12-16T21:58:43.481230+0000","last_peered":"2023-12-16T21:58:43.481230+0000","last_clean":"2023-12-16T21:58:43.481230+0000","last_became_active":"2023-12-16T21:58:41.563225+0000","last_became_peered":"2023-12-16T21:58:41.563225+0000","last_unstale":"2023-12-16T21:58:43.481230+0000","last_undegraded":"2023-12-16T21:58:43.481230+0000","last_fullsized":"2023-12-16T21:58:43.481230+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:43:03.724509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.482242+0000","last_change":"2023-12-16T21:58:41.558695+0000","last_active":"2023-12-16T21:58:43.482242+0000","last_peered":"2023-12-16T21:58:43.482242+0000","last_clean":"2023-12-16T21:58:43.482242+0000","last_became_active":"2023-12-16T21:58:41.557618+0000","last_became_peered":"2023-12-16T21:58:41.557618+0000","last_unstale":"2023-12-16T21:58:43.482242+0000","last_undegraded":"2023-12-16T21:58:43.482242+0000","last_fullsized":"2023-12-16T21:58:43.482242+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:47:31.519501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.482221+0000","last_change":"2023-12-16T21:58:41.558808+0000","last_active":"2023-12-16T21:58:43.482221+0000","last_peered":"2023-12-16T21:58:43.482221+0000","last_clean":"2023-12-16T21:58:43.482221+0000","last_became_active":"2023-12-16T21:58:41.557738+0000","last_became_peered":"2023-12-16T21:58:41.557738+0000","last_unstale":"2023-12-16T21:58:43.482221+0000","last_undegraded":"2023-12-16T21:58:43.482221+0000","last_fullsized":"2023-12-16T21:58:43.482221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:26:56.270290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.760694+0000","last_change":"2023-12-16T21:58:41.538625+0000","last_active":"2023-12-16T21:58:42.760694+0000","last_peered":"2023-12-16T21:58:42.760694+0000","last_clean":"2023-12-16T21:58:42.760694+0000","last_became_active":"2023-12-16T21:58:41.538242+0000","last_became_peered":"2023-12-16T21:58:41.538242+0000","last_unstale":"2023-12-16T21:58:42.760694+0000","last_undegraded":"2023-12-16T21:58:42.760694+0000","last_fullsized":"2023-12-16T21:58:42.760694+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:35:24.981673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.508477+0000","last_change":"2023-12-16T21:58:41.536012+0000","last_active":"2023-12-16T21:58:42.508477+0000","last_peered":"2023-12-16T21:58:42.508477+0000","last_clean":"2023-12-16T21:58:42.508477+0000","last_became_active":"2023-12-16T21:58:41.534720+0000","last_became_peered":"2023-12-16T21:58:41.534720+0000","last_unstale":"2023-12-16T21:58:42.508477+0000","last_undegraded":"2023-12-16T21:58:42.508477+0000","last_fullsized":"2023-12-16T21:58:42.508477+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:52:44.129490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.506897+0000","last_change":"2023-12-16T21:58:41.538617+0000","last_active":"2023-12-16T21:58:42.506897+0000","last_peered":"2023-12-16T21:58:42.506897+0000","last_clean":"2023-12-16T21:58:42.506897+0000","last_became_active":"2023-12-16T21:58:41.538302+0000","last_became_peered":"2023-12-16T21:58:41.538302+0000","last_unstale":"2023-12-16T21:58:42.506897+0000","last_undegraded":"2023-12-16T21:58:42.506897+0000","last_fullsized":"2023-12-16T21:58:42.506897+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:52:44.129490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.045986+0000","last_change":"2023-12-16T21:58:41.533065+0000","last_active":"2023-12-16T21:58:43.045986+0000","last_peered":"2023-12-16T21:58:43.045986+0000","last_clean":"2023-12-16T21:58:43.045986+0000","last_became_active":"2023-12-16T21:58:41.523266+0000","last_became_peered":"2023-12-16T21:58:41.523266+0000","last_unstale":"2023-12-16T21:58:43.045986+0000","last_undegraded":"2023-12-16T21:58:43.045986+0000","last_fullsized":"2023-12-16T21:58:43.045986+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:36:34.677954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.506438+0000","last_change":"2023-12-16T21:58:41.559160+0000","last_active":"2023-12-16T21:58:42.506438+0000","last_peered":"2023-12-16T21:58:42.506438+0000","last_clean":"2023-12-16T21:58:42.506438+0000","last_became_active":"2023-12-16T21:58:41.558797+0000","last_became_peered":"2023-12-16T21:58:41.558797+0000","last_unstale":"2023-12-16T21:58:42.506438+0000","last_undegraded":"2023-12-16T21:58:42.506438+0000","last_fullsized":"2023-12-16T21:58:42.506438+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:16:52.739167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.509224+0000","last_change":"2023-12-16T21:58:41.539435+0000","last_active":"2023-12-16T21:58:42.509224+0000","last_peered":"2023-12-16T21:58:42.509224+0000","last_clean":"2023-12-16T21:58:42.509224+0000","last_became_active":"2023-12-16T21:58:41.538726+0000","last_became_peered":"2023-12-16T21:58:41.538726+0000","last_unstale":"2023-12-16T21:58:42.509224+0000","last_undegraded":"2023-12-16T21:58:42.509224+0000","last_fullsized":"2023-12-16T21:58:42.509224+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:40:46.601255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.482630+0000","last_change":"2023-12-16T21:58:41.567355+0000","last_active":"2023-12-16T21:58:43.482630+0000","last_peered":"2023-12-16T21:58:43.482630+0000","last_clean":"2023-12-16T21:58:43.482630+0000","last_became_active":"2023-12-16T21:58:41.566633+0000","last_became_peered":"2023-12-16T21:58:41.566633+0000","last_unstale":"2023-12-16T21:58:43.482630+0000","last_undegraded":"2023-12-16T21:58:43.482630+0000","last_fullsized":"2023-12-16T21:58:43.482630+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:09:34.005549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481248+0000","last_change":"2023-12-16T21:58:41.542984+0000","last_active":"2023-12-16T21:58:43.481248+0000","last_peered":"2023-12-16T21:58:43.481248+0000","last_clean":"2023-12-16T21:58:43.481248+0000","last_became_active":"2023-12-16T21:58:41.537449+0000","last_became_peered":"2023-12-16T21:58:41.537449+0000","last_unstale":"2023-12-16T21:58:43.481248+0000","last_undegraded":"2023-12-16T21:58:43.481248+0000","last_fullsized":"2023-12-16T21:58:43.481248+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:26:27.870031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.509848+0000","last_change":"2023-12-16T21:58:41.526455+0000","last_active":"2023-12-16T21:58:42.509848+0000","last_peered":"2023-12-16T21:58:42.509848+0000","last_clean":"2023-12-16T21:58:42.509848+0000","last_became_active":"2023-12-16T21:58:41.525279+0000","last_became_peered":"2023-12-16T21:58:41.525279+0000","last_unstale":"2023-12-16T21:58:42.509848+0000","last_undegraded":"2023-12-16T21:58:42.509848+0000","last_fullsized":"2023-12-16T21:58:42.509848+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:51:06.436122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480656+0000","last_change":"2023-12-16T21:58:41.569730+0000","last_active":"2023-12-16T21:58:43.480656+0000","last_peered":"2023-12-16T21:58:43.480656+0000","last_clean":"2023-12-16T21:58:43.480656+0000","last_became_active":"2023-12-16T21:58:41.569191+0000","last_became_peered":"2023-12-16T21:58:41.569191+0000","last_unstale":"2023-12-16T21:58:43.480656+0000","last_undegraded":"2023-12-16T21:58:43.480656+0000","last_fullsized":"2023-12-16T21:58:43.480656+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:13:46.595747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.761071+0000","last_change":"2023-12-16T21:58:41.538665+0000","last_active":"2023-12-16T21:58:42.761071+0000","last_peered":"2023-12-16T21:58:42.761071+0000","last_clean":"2023-12-16T21:58:42.761071+0000","last_became_active":"2023-12-16T21:58:41.538214+0000","last_became_peered":"2023-12-16T21:58:41.538214+0000","last_unstale":"2023-12-16T21:58:42.761071+0000","last_undegraded":"2023-12-16T21:58:42.761071+0000","last_fullsized":"2023-12-16T21:58:42.761071+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:26:56.270290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"35'4","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.565590+0000","last_change":"2023-12-16T21:58:40.510463+0000","last_active":"2023-12-16T21:58:41.565590+0000","last_peered":"2023-12-16T21:58:41.565590+0000","last_clean":"2023-12-16T21:58:41.565590+0000","last_became_active":"2023-12-16T21:58:40.509883+0000","last_became_peered":"2023-12-16T21:58:40.509883+0000","last_unstale":"2023-12-16T21:58:41.565590+0000","last_undegraded":"2023-12-16T21:58:41.565590+0000","last_fullsized":"2023-12-16T21:58:41.565590+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:37:11.035240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.510364+0000","last_change":"2023-12-16T21:58:41.560269+0000","last_active":"2023-12-16T21:58:42.510364+0000","last_peered":"2023-12-16T21:58:42.510364+0000","last_clean":"2023-12-16T21:58:42.510364+0000","last_became_active":"2023-12-16T21:58:41.559547+0000","last_became_peered":"2023-12-16T21:58:41.559547+0000","last_unstale":"2023-12-16T21:58:42.510364+0000","last_undegraded":"2023-12-16T21:58:42.510364+0000","last_fullsized":"2023-12-16T21:58:42.510364+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:24:44.807447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"35'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.560758+0000","last_change":"2023-12-16T21:58:40.509857+0000","last_active":"2023-12-16T21:58:41.560758+0000","last_peered":"2023-12-16T21:58:41.560758+0000","last_clean":"2023-12-16T21:58:41.560758+0000","last_became_active":"2023-12-16T21:58:40.509381+0000","last_became_peered":"2023-12-16T21:58:40.509381+0000","last_unstale":"2023-12-16T21:58:41.560758+0000","last_undegraded":"2023-12-16T21:58:41.560758+0000","last_fullsized":"2023-12-16T21:58:41.560758+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:07:12.471926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.482547+0000","last_change":"2023-12-16T21:58:41.558156+0000","last_active":"2023-12-16T21:58:43.482547+0000","last_peered":"2023-12-16T21:58:43.482547+0000","last_clean":"2023-12-16T21:58:43.482547+0000","last_became_active":"2023-12-16T21:58:41.557404+0000","last_became_peered":"2023-12-16T21:58:41.557404+0000","last_unstale":"2023-12-16T21:58:43.482547+0000","last_undegraded":"2023-12-16T21:58:43.482547+0000","last_fullsized":"2023-12-16T21:58:43.482547+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:23:21.000061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"35'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.541784+0000","last_change":"2023-12-16T21:58:40.501596+0000","last_active":"2023-12-16T21:58:41.541784+0000","last_peered":"2023-12-16T21:58:41.541784+0000","last_clean":"2023-12-16T21:58:41.541784+0000","last_became_active":"2023-12-16T21:58:40.501319+0000","last_became_peered":"2023-12-16T21:58:40.501319+0000","last_unstale":"2023-12-16T21:58:41.541784+0000","last_undegraded":"2023-12-16T21:58:41.541784+0000","last_fullsized":"2023-12-16T21:58:41.541784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:23:19.996572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.506573+0000","last_change":"2023-12-16T21:58:41.558924+0000","last_active":"2023-12-16T21:58:42.506573+0000","last_peered":"2023-12-16T21:58:42.506573+0000","last_clean":"2023-12-16T21:58:42.506573+0000","last_became_active":"2023-12-16T21:58:41.558670+0000","last_became_peered":"2023-12-16T21:58:41.558670+0000","last_unstale":"2023-12-16T21:58:42.506573+0000","last_undegraded":"2023-12-16T21:58:42.506573+0000","last_fullsized":"2023-12-16T21:58:42.506573+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:23:21.000061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"35'2","reported_seq":14,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481374+0000","last_change":"2023-12-16T21:58:40.502942+0000","last_active":"2023-12-16T21:58:43.481374+0000","last_peered":"2023-12-16T21:58:43.481374+0000","last_clean":"2023-12-16T21:58:43.481374+0000","last_became_active":"2023-12-16T21:58:40.502586+0000","last_became_peered":"2023-12-16T21:58:40.502586+0000","last_unstale":"2023-12-16T21:58:43.481374+0000","last_undegraded":"2023-12-16T21:58:43.481374+0000","last_fullsized":"2023-12-16T21:58:43.481374+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:47:41.478553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481581+0000","last_change":"2023-12-16T21:58:41.560482+0000","last_active":"2023-12-16T21:58:43.481581+0000","last_peered":"2023-12-16T21:58:43.481581+0000","last_clean":"2023-12-16T21:58:43.481581+0000","last_became_active":"2023-12-16T21:58:41.559709+0000","last_became_peered":"2023-12-16T21:58:41.559709+0000","last_unstale":"2023-12-16T21:58:43.481581+0000","last_undegraded":"2023-12-16T21:58:43.481581+0000","last_fullsized":"2023-12-16T21:58:43.481581+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:31:47.797579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"35'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.567810+0000","last_change":"2023-12-16T21:58:40.516801+0000","last_active":"2023-12-16T21:58:41.567810+0000","last_peered":"2023-12-16T21:58:41.567810+0000","last_clean":"2023-12-16T21:58:41.567810+0000","last_became_active":"2023-12-16T21:58:40.515829+0000","last_became_peered":"2023-12-16T21:58:40.515829+0000","last_unstale":"2023-12-16T21:58:41.567810+0000","last_undegraded":"2023-12-16T21:58:41.567810+0000","last_fullsized":"2023-12-16T21:58:41.567810+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:07:12.471926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.508922+0000","last_change":"2023-12-16T21:58:41.536612+0000","last_active":"2023-12-16T21:58:42.508922+0000","last_peered":"2023-12-16T21:58:42.508922+0000","last_clean":"2023-12-16T21:58:42.508922+0000","last_became_active":"2023-12-16T21:58:41.535857+0000","last_became_peered":"2023-12-16T21:58:41.535857+0000","last_unstale":"2023-12-16T21:58:42.508922+0000","last_undegraded":"2023-12-16T21:58:42.508922+0000","last_fullsized":"2023-12-16T21:58:42.508922+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:13:46.595747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"35'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.566567+0000","last_change":"2023-12-16T21:58:40.501591+0000","last_active":"2023-12-16T21:58:41.566567+0000","last_peered":"2023-12-16T21:58:41.566567+0000","last_clean":"2023-12-16T21:58:41.566567+0000","last_became_active":"2023-12-16T21:58:40.500870+0000","last_became_peered":"2023-12-16T21:58:40.500870+0000","last_unstale":"2023-12-16T21:58:41.566567+0000","last_undegraded":"2023-12-16T21:58:41.566567+0000","last_fullsized":"2023-12-16T21:58:41.566567+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:41:53.369429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481276+0000","last_change":"2023-12-16T21:58:41.555269+0000","last_active":"2023-12-16T21:58:43.481276+0000","last_peered":"2023-12-16T21:58:43.481276+0000","last_clean":"2023-12-16T21:58:43.481276+0000","last_became_active":"2023-12-16T21:58:41.554461+0000","last_became_peered":"2023-12-16T21:58:41.554461+0000","last_unstale":"2023-12-16T21:58:43.481276+0000","last_undegraded":"2023-12-16T21:58:43.481276+0000","last_fullsized":"2023-12-16T21:58:43.481276+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:21:11.953787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.503944+0000","last_change":"2023-12-16T21:58:40.499232+0000","last_active":"2023-12-16T21:58:41.503944+0000","last_peered":"2023-12-16T21:58:41.503944+0000","last_clean":"2023-12-16T21:58:41.503944+0000","last_became_active":"2023-12-16T21:58:40.498853+0000","last_became_peered":"2023-12-16T21:58:40.498853+0000","last_unstale":"2023-12-16T21:58:41.503944+0000","last_undegraded":"2023-12-16T21:58:41.503944+0000","last_fullsized":"2023-12-16T21:58:41.503944+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:37:11.035240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.505722+0000","last_change":"2023-12-16T21:58:41.555264+0000","last_active":"2023-12-16T21:58:42.505722+0000","last_peered":"2023-12-16T21:58:42.505722+0000","last_clean":"2023-12-16T21:58:42.505722+0000","last_became_active":"2023-12-16T21:58:41.554969+0000","last_became_peered":"2023-12-16T21:58:41.554969+0000","last_unstale":"2023-12-16T21:58:42.505722+0000","last_undegraded":"2023-12-16T21:58:42.505722+0000","last_fullsized":"2023-12-16T21:58:42.505722+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:21:11.953787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.545336+0000","last_change":"2023-12-16T21:58:40.511738+0000","last_active":"2023-12-16T21:58:41.545336+0000","last_peered":"2023-12-16T21:58:41.545336+0000","last_clean":"2023-12-16T21:58:41.545336+0000","last_became_active":"2023-12-16T21:58:40.510752+0000","last_became_peered":"2023-12-16T21:58:40.510752+0000","last_unstale":"2023-12-16T21:58:41.545336+0000","last_undegraded":"2023-12-16T21:58:41.545336+0000","last_fullsized":"2023-12-16T21:58:41.545336+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:29:38.052590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.509911+0000","last_change":"2023-12-16T21:58:41.548394+0000","last_active":"2023-12-16T21:58:42.509911+0000","last_peered":"2023-12-16T21:58:42.509911+0000","last_clean":"2023-12-16T21:58:42.509911+0000","last_became_active":"2023-12-16T21:58:41.541550+0000","last_became_peered":"2023-12-16T21:58:41.541550+0000","last_unstale":"2023-12-16T21:58:42.509911+0000","last_undegraded":"2023-12-16T21:58:42.509911+0000","last_fullsized":"2023-12-16T21:58:42.509911+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:38:15.225162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.504442+0000","last_change":"2023-12-16T21:58:40.510523+0000","last_active":"2023-12-16T21:58:41.504442+0000","last_peered":"2023-12-16T21:58:41.504442+0000","last_clean":"2023-12-16T21:58:41.504442+0000","last_became_active":"2023-12-16T21:58:40.510085+0000","last_became_peered":"2023-12-16T21:58:40.510085+0000","last_unstale":"2023-12-16T21:58:41.504442+0000","last_undegraded":"2023-12-16T21:58:41.504442+0000","last_fullsized":"2023-12-16T21:58:41.504442+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:29:38.052590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.045802+0000","last_change":"2023-12-16T21:58:41.533587+0000","last_active":"2023-12-16T21:58:43.045802+0000","last_peered":"2023-12-16T21:58:43.045802+0000","last_clean":"2023-12-16T21:58:43.045802+0000","last_became_active":"2023-12-16T21:58:41.526528+0000","last_became_peered":"2023-12-16T21:58:41.526528+0000","last_unstale":"2023-12-16T21:58:43.045802+0000","last_undegraded":"2023-12-16T21:58:43.045802+0000","last_fullsized":"2023-12-16T21:58:43.045802+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:31:47.797579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.510974+0000","last_change":"2023-12-16T21:58:40.500571+0000","last_active":"2023-12-16T21:58:41.510974+0000","last_peered":"2023-12-16T21:58:41.510974+0000","last_clean":"2023-12-16T21:58:41.510974+0000","last_became_active":"2023-12-16T21:58:40.500226+0000","last_became_peered":"2023-12-16T21:58:40.500226+0000","last_unstale":"2023-12-16T21:58:41.510974+0000","last_undegraded":"2023-12-16T21:58:41.510974+0000","last_fullsized":"2023-12-16T21:58:41.510974+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:18:57.185826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.045555+0000","last_change":"2023-12-16T21:58:41.534388+0000","last_active":"2023-12-16T21:58:43.045555+0000","last_peered":"2023-12-16T21:58:43.045555+0000","last_clean":"2023-12-16T21:58:43.045555+0000","last_became_active":"2023-12-16T21:58:41.534075+0000","last_became_peered":"2023-12-16T21:58:41.534075+0000","last_unstale":"2023-12-16T21:58:43.045555+0000","last_undegraded":"2023-12-16T21:58:43.045555+0000","last_fullsized":"2023-12-16T21:58:43.045555+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:39.342482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"35'3","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.571697+0000","last_change":"2023-12-16T21:58:40.518810+0000","last_active":"2023-12-16T21:58:41.571697+0000","last_peered":"2023-12-16T21:58:41.571697+0000","last_clean":"2023-12-16T21:58:41.571697+0000","last_became_active":"2023-12-16T21:58:40.518226+0000","last_became_peered":"2023-12-16T21:58:40.518226+0000","last_unstale":"2023-12-16T21:58:41.571697+0000","last_undegraded":"2023-12-16T21:58:41.571697+0000","last_fullsized":"2023-12-16T21:58:41.571697+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:23:19.996572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.508689+0000","last_change":"2023-12-16T21:58:41.552959+0000","last_active":"2023-12-16T21:58:42.508689+0000","last_peered":"2023-12-16T21:58:42.508689+0000","last_clean":"2023-12-16T21:58:42.508689+0000","last_became_active":"2023-12-16T21:58:41.552301+0000","last_became_peered":"2023-12-16T21:58:41.552301+0000","last_unstale":"2023-12-16T21:58:42.508689+0000","last_undegraded":"2023-12-16T21:58:42.508689+0000","last_fullsized":"2023-12-16T21:58:42.508689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:46:20.127519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"35'6","reported_seq":18,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.571208+0000","last_change":"2023-12-16T21:58:40.498753+0000","last_active":"2023-12-16T21:58:41.571208+0000","last_peered":"2023-12-16T21:58:41.571208+0000","last_clean":"2023-12-16T21:58:41.571208+0000","last_became_active":"2023-12-16T21:58:40.498385+0000","last_became_peered":"2023-12-16T21:58:40.498385+0000","last_unstale":"2023-12-16T21:58:41.571208+0000","last_undegraded":"2023-12-16T21:58:41.571208+0000","last_fullsized":"2023-12-16T21:58:41.571208+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:21:10.950297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.506049+0000","last_change":"2023-12-16T21:58:41.540685+0000","last_active":"2023-12-16T21:58:42.506049+0000","last_peered":"2023-12-16T21:58:42.506049+0000","last_clean":"2023-12-16T21:58:42.506049+0000","last_became_active":"2023-12-16T21:58:41.540327+0000","last_became_peered":"2023-12-16T21:58:41.540327+0000","last_unstale":"2023-12-16T21:58:42.506049+0000","last_undegraded":"2023-12-16T21:58:42.506049+0000","last_fullsized":"2023-12-16T21:58:42.506049+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:47:31.519501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"35'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.568043+0000","last_change":"2023-12-16T21:58:40.511320+0000","last_active":"2023-12-16T21:58:41.568043+0000","last_peered":"2023-12-16T21:58:41.568043+0000","last_clean":"2023-12-16T21:58:41.568043+0000","last_became_active":"2023-12-16T21:58:40.510584+0000","last_became_peered":"2023-12-16T21:58:40.510584+0000","last_unstale":"2023-12-16T21:58:41.568043+0000","last_undegraded":"2023-12-16T21:58:41.568043+0000","last_fullsized":"2023-12-16T21:58:41.568043+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:18:57.185826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.045490+0000","last_change":"2023-12-16T21:58:41.521750+0000","last_active":"2023-12-16T21:58:43.045490+0000","last_peered":"2023-12-16T21:58:43.045490+0000","last_clean":"2023-12-16T21:58:43.045490+0000","last_became_active":"2023-12-16T21:58:41.521213+0000","last_became_peered":"2023-12-16T21:58:41.521213+0000","last_unstale":"2023-12-16T21:58:43.045490+0000","last_undegraded":"2023-12-16T21:58:43.045490+0000","last_fullsized":"2023-12-16T21:58:43.045490+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:38:15.225162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.513828+0000","last_change":"2023-12-16T21:58:40.503816+0000","last_active":"2023-12-16T21:58:41.513828+0000","last_peered":"2023-12-16T21:58:41.513828+0000","last_clean":"2023-12-16T21:58:41.513828+0000","last_became_active":"2023-12-16T21:58:40.503266+0000","last_became_peered":"2023-12-16T21:58:40.503266+0000","last_unstale":"2023-12-16T21:58:41.513828+0000","last_undegraded":"2023-12-16T21:58:41.513828+0000","last_fullsized":"2023-12-16T21:58:41.513828+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:51:05.432632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480847+0000","last_change":"2023-12-16T21:58:41.569609+0000","last_active":"2023-12-16T21:58:43.480847+0000","last_peered":"2023-12-16T21:58:43.480847+0000","last_clean":"2023-12-16T21:58:43.480847+0000","last_became_active":"2023-12-16T21:58:41.569127+0000","last_became_peered":"2023-12-16T21:58:41.569127+0000","last_unstale":"2023-12-16T21:58:43.480847+0000","last_undegraded":"2023-12-16T21:58:43.480847+0000","last_fullsized":"2023-12-16T21:58:43.480847+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:38:15.225162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.504309+0000","last_change":"2023-12-16T21:58:40.510225+0000","last_active":"2023-12-16T21:58:41.504309+0000","last_peered":"2023-12-16T21:58:41.504309+0000","last_clean":"2023-12-16T21:58:41.504309+0000","last_became_active":"2023-12-16T21:58:40.509920+0000","last_became_peered":"2023-12-16T21:58:40.509920+0000","last_unstale":"2023-12-16T21:58:41.504309+0000","last_undegraded":"2023-12-16T21:58:41.504309+0000","last_fullsized":"2023-12-16T21:58:41.504309+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:51:05.432632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.510193+0000","last_change":"2023-12-16T21:58:41.568629+0000","last_active":"2023-12-16T21:58:42.510193+0000","last_peered":"2023-12-16T21:58:42.510193+0000","last_clean":"2023-12-16T21:58:42.510193+0000","last_became_active":"2023-12-16T21:58:41.567918+0000","last_became_peered":"2023-12-16T21:58:41.567918+0000","last_unstale":"2023-12-16T21:58:42.510193+0000","last_undegraded":"2023-12-16T21:58:42.510193+0000","last_fullsized":"2023-12-16T21:58:42.510193+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:29:39.056080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"35'8","reported_seq":20,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.542267+0000","last_change":"2023-12-16T21:58:40.497006+0000","last_active":"2023-12-16T21:58:41.542267+0000","last_peered":"2023-12-16T21:58:41.542267+0000","last_clean":"2023-12-16T21:58:41.542267+0000","last_became_active":"2023-12-16T21:58:40.496662+0000","last_became_peered":"2023-12-16T21:58:40.496662+0000","last_unstale":"2023-12-16T21:58:41.542267+0000","last_undegraded":"2023-12-16T21:58:41.542267+0000","last_fullsized":"2023-12-16T21:58:41.542267+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:36:06.936793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.482948+0000","last_change":"2023-12-16T21:58:41.568237+0000","last_active":"2023-12-16T21:58:43.482948+0000","last_peered":"2023-12-16T21:58:43.482948+0000","last_clean":"2023-12-16T21:58:43.482948+0000","last_became_active":"2023-12-16T21:58:41.567691+0000","last_became_peered":"2023-12-16T21:58:41.567691+0000","last_unstale":"2023-12-16T21:58:43.482948+0000","last_undegraded":"2023-12-16T21:58:43.482948+0000","last_fullsized":"2023-12-16T21:58:43.482948+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:16:52.739167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"35'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.541936+0000","last_change":"2023-12-16T21:58:40.503698+0000","last_active":"2023-12-16T21:58:41.541936+0000","last_peered":"2023-12-16T21:58:41.541936+0000","last_clean":"2023-12-16T21:58:41.541936+0000","last_became_active":"2023-12-16T21:58:40.503064+0000","last_became_peered":"2023-12-16T21:58:40.503064+0000","last_unstale":"2023-12-16T21:58:41.541936+0000","last_undegraded":"2023-12-16T21:58:41.541936+0000","last_fullsized":"2023-12-16T21:58:41.541936+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:36:06.936793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.510013+0000","last_change":"2023-12-16T21:58:41.569234+0000","last_active":"2023-12-16T21:58:42.510013+0000","last_peered":"2023-12-16T21:58:42.510013+0000","last_clean":"2023-12-16T21:58:42.510013+0000","last_became_active":"2023-12-16T21:58:41.568554+0000","last_became_peered":"2023-12-16T21:58:41.568554+0000","last_unstale":"2023-12-16T21:58:42.510013+0000","last_undegraded":"2023-12-16T21:58:42.510013+0000","last_fullsized":"2023-12-16T21:58:42.510013+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:39.342482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"35'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.567327+0000","last_change":"2023-12-16T21:58:40.515564+0000","last_active":"2023-12-16T21:58:41.567327+0000","last_peered":"2023-12-16T21:58:41.567327+0000","last_clean":"2023-12-16T21:58:41.567327+0000","last_became_active":"2023-12-16T21:58:40.514839+0000","last_became_peered":"2023-12-16T21:58:40.514839+0000","last_unstale":"2023-12-16T21:58:41.567327+0000","last_undegraded":"2023-12-16T21:58:41.567327+0000","last_fullsized":"2023-12-16T21:58:41.567327+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:36:06.936793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.506185+0000","last_change":"2023-12-16T21:58:41.559790+0000","last_active":"2023-12-16T21:58:42.506185+0000","last_peered":"2023-12-16T21:58:42.506185+0000","last_clean":"2023-12-16T21:58:42.506185+0000","last_became_active":"2023-12-16T21:58:41.559451+0000","last_became_peered":"2023-12-16T21:58:41.559451+0000","last_unstale":"2023-12-16T21:58:42.506185+0000","last_undegraded":"2023-12-16T21:58:42.506185+0000","last_fullsized":"2023-12-16T21:58:42.506185+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:09:34.005549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.566766+0000","last_change":"2023-12-16T21:58:40.499987+0000","last_active":"2023-12-16T21:58:41.566766+0000","last_peered":"2023-12-16T21:58:41.566766+0000","last_clean":"2023-12-16T21:58:41.566766+0000","last_became_active":"2023-12-16T21:58:40.499198+0000","last_became_peered":"2023-12-16T21:58:40.499198+0000","last_unstale":"2023-12-16T21:58:41.566766+0000","last_undegraded":"2023-12-16T21:58:41.566766+0000","last_fullsized":"2023-12-16T21:58:41.566766+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:48:29.295675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481468+0000","last_change":"2023-12-16T21:58:41.544683+0000","last_active":"2023-12-16T21:58:43.481468+0000","last_peered":"2023-12-16T21:58:43.481468+0000","last_clean":"2023-12-16T21:58:43.481468+0000","last_became_active":"2023-12-16T21:58:41.544362+0000","last_became_peered":"2023-12-16T21:58:41.544362+0000","last_unstale":"2023-12-16T21:58:43.481468+0000","last_undegraded":"2023-12-16T21:58:43.481468+0000","last_fullsized":"2023-12-16T21:58:43.481468+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:42:05.207437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.540030+0000","last_change":"2023-12-16T21:58:40.503566+0000","last_active":"2023-12-16T21:58:41.540030+0000","last_peered":"2023-12-16T21:58:41.540030+0000","last_clean":"2023-12-16T21:58:41.540030+0000","last_became_active":"2023-12-16T21:58:40.502798+0000","last_became_peered":"2023-12-16T21:58:40.502798+0000","last_unstale":"2023-12-16T21:58:41.540030+0000","last_undegraded":"2023-12-16T21:58:41.540030+0000","last_fullsized":"2023-12-16T21:58:41.540030+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:37:11.035240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480781+0000","last_change":"2023-12-16T21:58:41.567440+0000","last_active":"2023-12-16T21:58:43.480781+0000","last_peered":"2023-12-16T21:58:43.480781+0000","last_clean":"2023-12-16T21:58:43.480781+0000","last_became_active":"2023-12-16T21:58:41.567135+0000","last_became_peered":"2023-12-16T21:58:41.567135+0000","last_unstale":"2023-12-16T21:58:43.480781+0000","last_undegraded":"2023-12-16T21:58:43.480781+0000","last_fullsized":"2023-12-16T21:58:43.480781+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:37:41.637920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"35'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.533034+0000","last_change":"2023-12-16T21:58:40.498021+0000","last_active":"2023-12-16T21:58:41.533034+0000","last_peered":"2023-12-16T21:58:41.533034+0000","last_clean":"2023-12-16T21:58:41.533034+0000","last_became_active":"2023-12-16T21:58:40.497676+0000","last_became_peered":"2023-12-16T21:58:40.497676+0000","last_unstale":"2023-12-16T21:58:41.533034+0000","last_undegraded":"2023-12-16T21:58:41.533034+0000","last_fullsized":"2023-12-16T21:58:41.533034+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:57.408305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481262+0000","last_change":"2023-12-16T21:58:41.532084+0000","last_active":"2023-12-16T21:58:43.481262+0000","last_peered":"2023-12-16T21:58:43.481262+0000","last_clean":"2023-12-16T21:58:43.481262+0000","last_became_active":"2023-12-16T21:58:41.529955+0000","last_became_peered":"2023-12-16T21:58:41.529955+0000","last_unstale":"2023-12-16T21:58:43.481262+0000","last_undegraded":"2023-12-16T21:58:43.481262+0000","last_fullsized":"2023-12-16T21:58:43.481262+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:07:34.643469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.510332+0000","last_change":"2023-12-16T21:58:40.502158+0000","last_active":"2023-12-16T21:58:41.510332+0000","last_peered":"2023-12-16T21:58:41.510332+0000","last_clean":"2023-12-16T21:58:41.510332+0000","last_became_active":"2023-12-16T21:58:40.501224+0000","last_became_peered":"2023-12-16T21:58:40.501224+0000","last_unstale":"2023-12-16T21:58:41.510332+0000","last_undegraded":"2023-12-16T21:58:41.510332+0000","last_fullsized":"2023-12-16T21:58:41.510332+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:07:12.471926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480758+0000","last_change":"2023-12-16T21:58:41.520385+0000","last_active":"2023-12-16T21:58:43.480758+0000","last_peered":"2023-12-16T21:58:43.480758+0000","last_clean":"2023-12-16T21:58:43.480758+0000","last_became_active":"2023-12-16T21:58:41.520005+0000","last_became_peered":"2023-12-16T21:58:41.520005+0000","last_unstale":"2023-12-16T21:58:43.480758+0000","last_undegraded":"2023-12-16T21:58:43.480758+0000","last_fullsized":"2023-12-16T21:58:43.480758+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:34:53.394229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.542182+0000","last_change":"2023-12-16T21:58:40.503347+0000","last_active":"2023-12-16T21:58:41.542182+0000","last_peered":"2023-12-16T21:58:41.542182+0000","last_clean":"2023-12-16T21:58:41.542182+0000","last_became_active":"2023-12-16T21:58:40.502397+0000","last_became_peered":"2023-12-16T21:58:40.502397+0000","last_unstale":"2023-12-16T21:58:41.542182+0000","last_undegraded":"2023-12-16T21:58:41.542182+0000","last_fullsized":"2023-12-16T21:58:41.542182+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:57.408305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.480699+0000","last_change":"2023-12-16T21:58:41.546291+0000","last_active":"2023-12-16T21:58:43.480699+0000","last_peered":"2023-12-16T21:58:43.480699+0000","last_clean":"2023-12-16T21:58:43.480699+0000","last_became_active":"2023-12-16T21:58:41.544803+0000","last_became_peered":"2023-12-16T21:58:41.544803+0000","last_unstale":"2023-12-16T21:58:43.480699+0000","last_undegraded":"2023-12-16T21:58:43.480699+0000","last_fullsized":"2023-12-16T21:58:43.480699+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:40:46.601255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.517708+0000","last_change":"2023-12-16T21:58:40.519887+0000","last_active":"2023-12-16T21:58:41.517708+0000","last_peered":"2023-12-16T21:58:41.517708+0000","last_clean":"2023-12-16T21:58:41.517708+0000","last_became_active":"2023-12-16T21:58:40.518760+0000","last_became_peered":"2023-12-16T21:58:40.518760+0000","last_unstale":"2023-12-16T21:58:41.517708+0000","last_undegraded":"2023-12-16T21:58:41.517708+0000","last_fullsized":"2023-12-16T21:58:41.517708+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:24:43.803958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481217+0000","last_change":"2023-12-16T21:58:41.541134+0000","last_active":"2023-12-16T21:58:43.481217+0000","last_peered":"2023-12-16T21:58:43.481217+0000","last_clean":"2023-12-16T21:58:43.481217+0000","last_became_active":"2023-12-16T21:58:41.540881+0000","last_became_peered":"2023-12-16T21:58:41.540881+0000","last_unstale":"2023-12-16T21:58:43.481217+0000","last_undegraded":"2023-12-16T21:58:43.481217+0000","last_fullsized":"2023-12-16T21:58:43.481217+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:30:06.360464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"35'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.564647+0000","last_change":"2023-12-16T21:58:40.510461+0000","last_active":"2023-12-16T21:58:41.564647+0000","last_peered":"2023-12-16T21:58:41.564647+0000","last_clean":"2023-12-16T21:58:41.564647+0000","last_became_active":"2023-12-16T21:58:40.509347+0000","last_became_peered":"2023-12-16T21:58:40.509347+0000","last_unstale":"2023-12-16T21:58:41.564647+0000","last_undegraded":"2023-12-16T21:58:41.564647+0000","last_fullsized":"2023-12-16T21:58:41.564647+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:57.408305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.510115+0000","last_change":"2023-12-16T21:58:41.539161+0000","last_active":"2023-12-16T21:58:42.510115+0000","last_peered":"2023-12-16T21:58:42.510115+0000","last_clean":"2023-12-16T21:58:42.510115+0000","last_became_active":"2023-12-16T21:58:41.538436+0000","last_became_peered":"2023-12-16T21:58:41.538436+0000","last_unstale":"2023-12-16T21:58:42.510115+0000","last_undegraded":"2023-12-16T21:58:42.510115+0000","last_fullsized":"2023-12-16T21:58:42.510115+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:53:20.775769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"36'1","reported_seq":13,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.761118+0000","last_change":"2023-12-16T21:58:40.495901+0000","last_active":"2023-12-16T21:58:42.761118+0000","last_peered":"2023-12-16T21:58:42.761118+0000","last_clean":"2023-12-16T21:58:42.761118+0000","last_became_active":"2023-12-16T21:58:40.495531+0000","last_became_peered":"2023-12-16T21:58:40.495531+0000","last_unstale":"2023-12-16T21:58:42.761118+0000","last_undegraded":"2023-12-16T21:58:42.761118+0000","last_fullsized":"2023-12-16T21:58:42.761118+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:37:11.035240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.760934+0000","last_change":"2023-12-16T21:58:41.526532+0000","last_active":"2023-12-16T21:58:42.760934+0000","last_peered":"2023-12-16T21:58:42.760934+0000","last_clean":"2023-12-16T21:58:42.760934+0000","last_became_active":"2023-12-16T21:58:41.526246+0000","last_became_peered":"2023-12-16T21:58:41.526246+0000","last_unstale":"2023-12-16T21:58:42.760934+0000","last_undegraded":"2023-12-16T21:58:42.760934+0000","last_fullsized":"2023-12-16T21:58:42.760934+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:21:11.953787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.519039+0000","last_change":"2023-12-16T21:58:40.501386+0000","last_active":"2023-12-16T21:58:41.519039+0000","last_peered":"2023-12-16T21:58:41.519039+0000","last_clean":"2023-12-16T21:58:41.519039+0000","last_became_active":"2023-12-16T21:58:40.500462+0000","last_became_peered":"2023-12-16T21:58:40.500462+0000","last_unstale":"2023-12-16T21:58:41.519039+0000","last_undegraded":"2023-12-16T21:58:41.519039+0000","last_fullsized":"2023-12-16T21:58:41.519039+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:24:43.803958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.509693+0000","last_change":"2023-12-16T21:58:41.527850+0000","last_active":"2023-12-16T21:58:42.509693+0000","last_peered":"2023-12-16T21:58:42.509693+0000","last_clean":"2023-12-16T21:58:42.509693+0000","last_became_active":"2023-12-16T21:58:41.522791+0000","last_became_peered":"2023-12-16T21:58:41.522791+0000","last_unstale":"2023-12-16T21:58:42.509693+0000","last_undegraded":"2023-12-16T21:58:42.509693+0000","last_fullsized":"2023-12-16T21:58:42.509693+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:58.411795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.512109+0000","last_change":"2023-12-16T21:58:40.509180+0000","last_active":"2023-12-16T21:58:41.512109+0000","last_peered":"2023-12-16T21:58:41.512109+0000","last_clean":"2023-12-16T21:58:41.512109+0000","last_became_active":"2023-12-16T21:58:40.508886+0000","last_became_peered":"2023-12-16T21:58:40.508886+0000","last_unstale":"2023-12-16T21:58:41.512109+0000","last_undegraded":"2023-12-16T21:58:41.512109+0000","last_fullsized":"2023-12-16T21:58:41.512109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:51:05.432632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:42.760927+0000","last_change":"2023-12-16T21:58:41.554480+0000","last_active":"2023-12-16T21:58:42.760927+0000","last_peered":"2023-12-16T21:58:42.760927+0000","last_clean":"2023-12-16T21:58:42.760927+0000","last_became_active":"2023-12-16T21:58:41.554187+0000","last_became_peered":"2023-12-16T21:58:41.554187+0000","last_unstale":"2023-12-16T21:58:42.760927+0000","last_undegraded":"2023-12-16T21:58:42.760927+0000","last_fullsized":"2023-12-16T21:58:42.760927+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:24:44.807447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.513995+0000","last_change":"2023-12-16T21:58:40.515885+0000","last_active":"2023-12-16T21:58:41.513995+0000","last_peered":"2023-12-16T21:58:41.513995+0000","last_clean":"2023-12-16T21:58:41.513995+0000","last_became_active":"2023-12-16T21:58:40.515446+0000","last_became_peered":"2023-12-16T21:58:40.515446+0000","last_unstale":"2023-12-16T21:58:41.513995+0000","last_undegraded":"2023-12-16T21:58:41.513995+0000","last_fullsized":"2023-12-16T21:58:41.513995+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:29:38.052590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.044597+0000","last_change":"2023-12-16T21:58:41.533087+0000","last_active":"2023-12-16T21:58:43.044597+0000","last_peered":"2023-12-16T21:58:43.044597+0000","last_clean":"2023-12-16T21:58:43.044597+0000","last_became_active":"2023-12-16T21:58:41.530760+0000","last_became_peered":"2023-12-16T21:58:41.530760+0000","last_unstale":"2023-12-16T21:58:43.044597+0000","last_undegraded":"2023-12-16T21:58:43.044597+0000","last_fullsized":"2023-12-16T21:58:43.044597+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:59:35.025704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.542425+0000","last_change":"2023-12-16T21:58:40.516094+0000","last_active":"2023-12-16T21:58:41.542425+0000","last_peered":"2023-12-16T21:58:41.542425+0000","last_clean":"2023-12-16T21:58:41.542425+0000","last_became_active":"2023-12-16T21:58:40.515413+0000","last_became_peered":"2023-12-16T21:58:40.515413+0000","last_unstale":"2023-12-16T21:58:41.542425+0000","last_undegraded":"2023-12-16T21:58:41.542425+0000","last_fullsized":"2023-12-16T21:58:41.542425+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:51:05.432632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481163+0000","last_change":"2023-12-16T21:58:41.558226+0000","last_active":"2023-12-16T21:58:43.481163+0000","last_peered":"2023-12-16T21:58:43.481163+0000","last_clean":"2023-12-16T21:58:43.481163+0000","last_became_active":"2023-12-16T21:58:41.557502+0000","last_became_peered":"2023-12-16T21:58:41.557502+0000","last_unstale":"2023-12-16T21:58:43.481163+0000","last_undegraded":"2023-12-16T21:58:43.481163+0000","last_fullsized":"2023-12-16T21:58:43.481163+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:24:44.807447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.504131+0000","last_change":"2023-12-16T21:58:40.510143+0000","last_active":"2023-12-16T21:58:41.504131+0000","last_peered":"2023-12-16T21:58:41.504131+0000","last_clean":"2023-12-16T21:58:41.504131+0000","last_became_active":"2023-12-16T21:58:40.509825+0000","last_became_peered":"2023-12-16T21:58:40.509825+0000","last_unstale":"2023-12-16T21:58:41.504131+0000","last_undegraded":"2023-12-16T21:58:41.504131+0000","last_fullsized":"2023-12-16T21:58:41.504131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:28:57.408305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:58:43.481102+0000","last_change":"2023-12-16T21:58:41.543489+0000","last_active":"2023-12-16T21:58:43.481102+0000","last_peered":"2023-12-16T21:58:43.481102+0000","last_clean":"2023-12-16T21:58:43.481102+0000","last_became_active":"2023-12-16T21:58:41.537804+0000","last_became_peered":"2023-12-16T21:58:41.537804+0000","last_unstale":"2023-12-16T21:58:43.481102+0000","last_undegraded":"2023-12-16T21:58:43.481102+0000","last_fullsized":"2023-12-16T21:58:43.481102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":34,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:40.470714+0000","last_clean_scrub_stamp":"2023-12-16T21:58:40.470714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:53:20.775769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.510690+0000","last_change":"2023-12-16T21:58:40.516743+0000","last_active":"2023-12-16T21:58:41.510690+0000","last_peered":"2023-12-16T21:58:41.510690+0000","last_clean":"2023-12-16T21:58:41.510690+0000","last_became_active":"2023-12-16T21:58:40.515578+0000","last_became_peered":"2023-12-16T21:58:40.515578+0000","last_unstale":"2023-12-16T21:58:41.510690+0000","last_undegraded":"2023-12-16T21:58:41.510690+0000","last_fullsized":"2023-12-16T21:58:41.510690+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:58:39.467225+0000","last_clean_scrub_stamp":"2023-12-16T21:58:39.467225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:18:57.185826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":174,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-16T21:58:41.513993+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-16T21:58:41.513993+0000","last_peered":"2023-12-16T21:58:41.513993+0000","last_clean":"2023-12-16T21:58:41.513993+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-16T21:58:41.513993+0000","last_undegraded":"2023-12-16T21:58:41.513993+0000","last_fullsized":"2023-12-16T21:58:41.513993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719476755,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":995328,"data_stored":746113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.874,"15min":0.874},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1.0669999999999999}]},{"osd":1,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.94499999999999995}]},{"osd":3,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.89100000000000001}]},{"osd":4,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5509999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.042}]},{"osd":5,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.0049999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:58:32 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.9710000000000001,"5min":1.9710000000000001,"15min":1.9710000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.3120000000000001}]}]},{"osd":6,"up_from":16,"seq":68719476754,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":156271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":1.071},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":3.7000000000000002,"5min":3.7000000000000002,"15min":3.7000000000000002},"last":0.78000000000000003}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.524,"5min":1.524,"15min":1.524},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.93100000000000005}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.88100000000000001}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":1.1719999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":5.5620000000000003,"5min":5.5620000000000003,"15min":5.5620000000000003},"last":0.98299999999999998}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":4.3620000000000001,"5min":4.3620000000000001,"15min":4.3620000000000001},"last":1.1439999999999999}]}]},{"osd":5,"up_from":16,"seq":68719476755,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":156327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.92400000000000004}]},{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":3.0299999999999998,"5min":3.0299999999999998,"15min":3.0299999999999998},"last":0.95599999999999996}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.81799999999999995}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.0649999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":3.2050000000000001,"5min":3.2050000000000001,"15min":3.2050000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.028}]},{"osd":6,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":2.6400000000000001,"5min":2.6400000000000001,"15min":2.6400000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.254,"5min":3.254,"15min":3.254},"last":1.0980000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.512,"5min":2.512,"15min":2.512},"last":1.151},{"interface":"front","average":{"1min":1.325,"5min":1.325,"15min":1.325},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":4.0129999999999999,"5min":4.0129999999999999,"15min":4.0129999999999999},"last":1.1830000000000001}]}]},{"osd":4,"up_from":16,"seq":68719476753,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":155759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.877}]},{"osd":1,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.90500000000000003}]},{"osd":2,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.80300000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":1.0029999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.056,"5min":1.056,"15min":1.056},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.6679999999999999,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.077,"15min":1.077},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.0609999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.032},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.95399999999999996}]}]},{"osd":3,"up_from":15,"seq":64424509460,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":157289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.70999999999999996}]},{"osd":1,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.91100000000000003}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.746},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.625}]},{"osd":4,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.054},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.22,"5min":1.22,"15min":1.22},"last":0.95899999999999996}]},{"osd":5,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.101},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.135}]},{"osd":6,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.175},{"interface":"front","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.244}]},{"osd":7,"last update":"Sat Dec 16 21:58:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.258,"5min":2.258,"15min":2.258},"last":1.212},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":1.2709999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509458,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":413696,"data_stored":157289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.746},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.998}]},{"osd":2,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":1.044}]},{"osd":3,"last update":"Sat Dec 16 21:58:29 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":1.103}]},{"osd":4,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.153},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.89900000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":1.254},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":1.3240000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":1.375},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.3080000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:58:31 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.349,"5min":1.349,"15min":1.349},"last":1.423},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":1.544}]}]},{"osd":0,"up_from":15,"seq":64424509458,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":991232,"data_stored":746023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.81200000000000006}]},{"osd":3,"last update":"Sat Dec 16 21:58:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":1.1419999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.96099999999999997}]},{"osd":5,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":1.244},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.1850000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.016},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":2.081,"5min":2.081,"15min":2.081},"last":1.331}]},{"osd":7,"last update":"Sat Dec 16 21:58:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.099},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":1.2190000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542162,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":155725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.42799999999999999}]},{"osd":1,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.554,"5min":1.554,"15min":1.554},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":0.71099999999999997}]},{"osd":5,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":0.81200000000000006}]},{"osd":6,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.94999999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:58:37 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.899,"5min":1.899,"15min":1.899},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.86499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:58:49.456 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-16T21:58:49.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-16T21:58:49.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff3d26640 1 Processor -- start 2023-12-16T21:58:49.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff3d26640 1 -- start start 2023-12-16T21:58:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff3d26640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fcfec143d10 0x7fcfec15f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff3d26640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 0x7fcfec0b4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff3d26640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfec1446f0 con 0x7fcfec070480 2023-12-16T21:58:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff3d26640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfec071590 con 0x7fcfec143d10 2023-12-16T21:58:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff3d26640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec0b4cc0 con 0x7fcfec0b42d0 2023-12-16T21:58:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff12c1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcfec070480 0x7fcfec15b5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50076/0 (socket says 172.21.15.28:50076) 2023-12-16T21:58:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff12c1640 1 -- 172.21.15.28:0/1387758 learned_addr learned my addr 172.21.15.28:0/1387758 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:58:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170105557 0 0) 0x7fcfec1446f0 con 0x7fcfec070480 2023-12-16T21:58:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff22c3640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 0x7fcfec0b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff1ac2640 1 --2- 172.21.15.28:0/1387758 >> v2:172.21.15.122:3300/0 conn(0x7fcfec143d10 0x7fcfec15f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd8003530 con 0x7fcfec070480 2023-12-16T21:58:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3367031939 0 0) 0x7fcfd8003530 con 0x7fcfec070480 2023-12-16T21:58:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 msgr2=0x7fcfec0b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 0x7fcfec0b4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 >> v2:172.21.15.122:3300/0 conn(0x7fcfec143d10 msgr2=0x7fcfec15f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 --2- 172.21.15.28:0/1387758 >> v2:172.21.15.122:3300/0 conn(0x7fcfec143d10 0x7fcfec15f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec070910 con 0x7fcfec070480 2023-12-16T21:58:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fcfdc001680 con 0x7fcfec070480 2023-12-16T21:58:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff1ac2640 1 --2- 172.21.15.28:0/1387758 >> v2:172.21.15.122:3300/0 conn(0x7fcfec143d10 0x7fcfec15f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 >> v1:172.21.15.28:6789/0 conn(0x7fcfec070480 legacy=0x7fcfec15b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd8003980 0x7fcfd8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd80047d0 0x7fcfd8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 0x7fcfd8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec071590 con 0x7fcfd80047d0 2023-12-16T21:58:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec0b4cc0 con 0x7fcfd8003980 2023-12-16T21:58:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.621+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd800ba70 con 0x7fcfec0b42d0 2023-12-16T21:58:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff12c1640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd80047d0 0x7fcfd8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff1ac2640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd8003980 0x7fcfd8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff22c3640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 0x7fcfd8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff12c1640 1 -- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 msgr2=0x7fcfd8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:49.634 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff12c1640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 0x7fcfd8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff12c1640 1 -- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd8003980 msgr2=0x7fcfd8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff12c1640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd8003980 0x7fcfd8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff12c1640 1 -- 172.21.15.28:0/1387758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd8003530 con 0x7fcfd80047d0 2023-12-16T21:58:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff22c3640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b42d0 0x7fcfd8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff1ac2640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd8003980 0x7fcfd8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff12c1640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd80047d0 0x7fcfd8004c50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc001ec0 tx=0x7fcfdc01cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a005152adb8298 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc0058e0 con 0x7fcfd80047d0 2023-12-16T21:58:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfdc005a40 con 0x7fcfd80047d0 2023-12-16T21:58:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff0ac0640 1 -- 172.21.15.28:0/1387758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc005dd0 con 0x7fcfd80047d0 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff3d26640 1 -- 172.21.15.28:0/1387758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd80047d0 msgr2=0x7fcfd8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff3d26640 1 --2- 172.21.15.28:0/1387758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd80047d0 0x7fcfd8004c50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc001ec0 tx=0x7fcfdc01cb40 comp rx=0 tx=0).stop 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff22c3640 1 -- 172.21.15.28:0/1387758 reap_dead start 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff3d26640 1 -- 172.21.15.28:0/1387758 shutdown_connections 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff3d26640 1 -- 172.21.15.28:0/1387758 >> 172.21.15.28:0/1387758 conn(0x7fcfec0b0cc0 msgr2=0x7fcfec0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff3d26640 1 -- 172.21.15.28:0/1387758 shutdown_connections 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.625+0000 7fcff3d26640 1 -- 172.21.15.28:0/1387758 wait complete. 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 Processor -- start 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 -- start start 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd80047d0 0x7fcfec15a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b3b60 0x7fcfec1608d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec139a90 0x7fcfec139f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec13a440 con 0x7fcfec139a90 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec161240 con 0x7fcfd80047d0 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec161520 con 0x7fcfec0b3b60 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec139a90 0x7fcfec139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec139a90 0x7fcfec139f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36176/0 (socket says 172.21.15.28:36176) 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 -- 172.21.15.28:0/1663581531 learned_addr learned my addr 172.21.15.28:0/1663581531 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff12c1640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b3b60 0x7fcfec1608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff1ac2640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd80047d0 0x7fcfec15a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 -- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b3b60 msgr2=0x7fcfec1608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b3b60 0x7fcfec1608d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 -- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd80047d0 msgr2=0x7fcfec15a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd80047d0 0x7fcfec15a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 -- 172.21.15.28:0/1663581531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec161800 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff12c1640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b3b60 0x7fcfec1608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff1ac2640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd80047d0 0x7fcfec15a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff22c3640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec139a90 0x7fcfec139f00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0672a0 tx=0x7fcfe400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcfe27fc640 1 -- 172.21.15.28:0/1663581531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4017400 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcfe27fc640 1 -- 172.21.15.28:0/1663581531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe4011040 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcfe27fc640 1 -- 172.21.15.28:0/1663581531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4017780 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec161ae0 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec15a620 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec136260 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcfe27fc640 1 -- 172.21.15.28:0/1663581531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fcfe40203b0 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.629+0000 7fcfe27fc640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcfd00664a0 0x7fcfd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.633+0000 7fcfe27fc640 1 -- 172.21.15.28:0/1663581531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfe4083fe0 con 0x7fcfec139a90 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.633+0000 7fcff1ac2640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcfd00664a0 0x7fcfd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.633+0000 7fcff1ac2640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcfd00664a0 0x7fcfd0068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c850 tx=0x7fcfe8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.637+0000 7fcfe27fc640 1 -- 172.21.15.28:0/1663581531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fcfe4051460 con 0x7fcfec139a90 2023-12-16T21:58:49.852 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.849+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fcfec069440 con 0x7fcfec139a90 2023-12-16T21:58:49.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcfe27fc640 1 -- 172.21.15.28:0/1663581531 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fcfe4054a80 con 0x7fcfec139a90 2023-12-16T21:58:49.856 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2023-12-16T21:58:49.856 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAeHX5l+MjmGRAAJRtEVtccO6nez+DbuubEww== 2023-12-16T21:58:49.856 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2023-12-16T21:58:49.856 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2023-12-16T21:58:49.856 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-12-16T21:58:49.856 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2023-12-16T21:58:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcfd00664a0 msgr2=0x7fcfd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcfd00664a0 0x7fcfd0068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c850 tx=0x7fcfe8008040 comp rx=0 tx=0).stop 2023-12-16T21:58:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec139a90 msgr2=0x7fcfec139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec139a90 0x7fcfec139f00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0672a0 tx=0x7fcfe400f4a0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 shutdown_connections 2023-12-16T21:58:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fcfd00664a0 0x7fcfd0068960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfec139a90 0x7fcfec139f00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fcfec0b3b60 0x7fcfec1608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 --2- 172.21.15.28:0/1663581531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcfd80047d0 0x7fcfec15a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 >> 172.21.15.28:0/1663581531 conn(0x7fcfec0b0cc0 msgr2=0x7fcfec15ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 shutdown_connections 2023-12-16T21:58:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.853+0000 7fcff3d26640 1 -- 172.21.15.28:0/1663581531 wait complete. 2023-12-16T21:58:49.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:58:50.000 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f331896d640 1 Processor -- start 2023-12-16T21:58:50.000 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f331896d640 1 -- start start 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f331896d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314144840 0x7f331406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f331896d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3314072180 0x7f331406c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f331896d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33140710e0 con 0x7f3314070460 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f331896d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331415f010 con 0x7f3314072180 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f331896d640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33140713c0 con 0x7f3314144840 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f3311d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3314072180 0x7f331406c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f3311d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3314072180 0x7f331406c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36190/0 (socket says 172.21.15.28:36190) 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f3311d9b640 1 -- 172.21.15.28:0/1271014542 learned_addr learned my addr 172.21.15.28:0/1271014542 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f331259c640 1 --2- 172.21.15.28:0/1271014542 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314144840 0x7f331406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f3311d9b640 1 -- 172.21.15.28:0/1271014542 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314144840 msgr2=0x7f331406c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f3311d9b640 1 --2- 172.21.15.28:0/1271014542 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314144840 0x7f331406c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f3311d9b640 1 -- 172.21.15.28:0/1271014542 >> v1:172.21.15.122:6789/0 conn(0x7f3314070460 legacy=0x7f33140708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.004 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:49.997+0000 7f3311d9b640 1 -- 172.21.15.28:0/1271014542 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33140716a0 con 0x7f3314072180 2023-12-16T21:58:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f3311d9b640 1 --2- 172.21.15.28:0/1271014542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314072180 0x7f331406c800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3308009870 tx=0x7f330801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34de203072aed90c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331159a640 1 -- 172.21.15.28:0/1271014542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080aa670 con 0x7f3314072180 2023-12-16T21:58:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331159a640 1 -- 172.21.15.28:0/1271014542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33080aa7d0 con 0x7f3314072180 2023-12-16T21:58:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331159a640 1 -- 172.21.15.28:0/1271014542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080aab60 con 0x7f3314072180 2023-12-16T21:58:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331896d640 1 -- 172.21.15.28:0/1271014542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314072180 msgr2=0x7f331406c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331896d640 1 --2- 172.21.15.28:0/1271014542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314072180 0x7f331406c800 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3308009870 tx=0x7f330801c510 comp rx=0 tx=0).stop 2023-12-16T21:58:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331896d640 1 -- 172.21.15.28:0/1271014542 shutdown_connections 2023-12-16T21:58:50.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331896d640 1 --2- 172.21.15.28:0/1271014542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314072180 0x7f331406c800 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331896d640 1 --2- 172.21.15.28:0/1271014542 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314144840 0x7f331406c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331896d640 1 -- 172.21.15.28:0/1271014542 >> 172.21.15.28:0/1271014542 conn(0x7f331406a530 msgr2=0x7f331406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331896d640 1 -- 172.21.15.28:0/1271014542 shutdown_connections 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.001+0000 7f331896d640 1 -- 172.21.15.28:0/1271014542 wait complete. 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 Processor -- start 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 -- start start 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3314070460 0x7f331415a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314072180 0x7f3314160140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314144840 0x7f3314159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3314160d40 con 0x7f3314144840 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3314142da0 con 0x7f3314070460 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3314142220 con 0x7f3314072180 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314144840 0x7f3314159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314144840 0x7f3314159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36204/0 (socket says 172.21.15.28:36204) 2023-12-16T21:58:50.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 -- 172.21.15.28:0/4222874923 learned_addr learned my addr 172.21.15.28:0/4222874923 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3311d9b640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314072180 0x7f3314160140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331259c640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3314070460 0x7f331415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 -- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314072180 msgr2=0x7f3314160140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314072180 0x7f3314160140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 -- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3314070460 msgr2=0x7f331415a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3314070460 0x7f331415a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 -- 172.21.15.28:0/4222874923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33080094d0 con 0x7f3314144840 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3312d9d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314144840 0x7f3314159cd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f330c00c770 tx=0x7f330c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3302ffd640 1 -- 172.21.15.28:0/4222874923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c003b80 con 0x7f3314144840 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3302ffd640 1 -- 172.21.15.28:0/4222874923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c003ce0 con 0x7f3314144840 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3314142560 con 0x7f3314144840 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3302ffd640 1 -- 172.21.15.28:0/4222874923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c00aa90 con 0x7f3314144840 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331415ae80 con 0x7f3314144840 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331413eb90 con 0x7f3314144840 2023-12-16T21:58:50.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.005+0000 7f3302ffd640 1 -- 172.21.15.28:0/4222874923 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f330c00abf0 con 0x7f3314144840 2023-12-16T21:58:50.014 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.009+0000 7f3302ffd640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f32f4066470 0x7f32f4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.014 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.009+0000 7f3302ffd640 1 -- 172.21.15.28:0/4222874923 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f330c0846b0 con 0x7f3314144840 2023-12-16T21:58:50.014 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.009+0000 7f331259c640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f32f4066470 0x7f32f4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.015 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.009+0000 7f331259c640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f32f4066470 0x7f32f4068930 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3304009a50 tx=0x7f3304008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.015 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.013+0000 7f3302ffd640 1 -- 172.21.15.28:0/4222874923 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f330c051bc0 con 0x7f3314144840 2023-12-16T21:58:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.197+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f331406d170 con 0x7f3314144840 2023-12-16T21:58:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.197+0000 7f3302ffd640 1 -- 172.21.15.28:0/4222874923 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f330c0551e0 con 0x7f3314144840 2023-12-16T21:58:50.203 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:58:50.203 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/992095456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":992095456},{"type":"v1","addr":"172.21.15.122:6835","nonce":992095456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3174619236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3174619236},{"type":"v1","addr":"172.21.15.122:6837","nonce":3174619236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4691,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3342817962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3342817962},{"type":"v1","addr":"172.21.15.28:6835","nonce":3342817962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:58:40.841493+0000","modified":"2023-12-16T21:58:42.871930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5193},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5193":{"gid":5193,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/924398795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":924398795},{"type":"v1","addr":"172.21.15.28:6833","nonce":924398795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:58:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 15 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f32f4066470 msgr2=0x7f32f4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f32f4066470 0x7f32f4068930 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3304009a50 tx=0x7f3304008040 comp rx=0 tx=0).stop 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314144840 msgr2=0x7f3314159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314144840 0x7f3314159cd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f330c00c770 tx=0x7f330c00cc30 comp rx=0 tx=0).stop 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 shutdown_connections 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f32f4066470 0x7f32f4068930 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3314144840 0x7f3314159cd0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3314072180 0x7f3314160140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 --2- 172.21.15.28:0/4222874923 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3314070460 0x7f331415a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 >> 172.21.15.28:0/4222874923 conn(0x7f331406a530 msgr2=0x7f3314165140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 shutdown_connections 2023-12-16T21:58:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.201+0000 7f331896d640 1 -- 172.21.15.28:0/4222874923 wait complete. 2023-12-16T21:58:50.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-16T21:58:40.841493+0000', 'modified': '2023-12-16T21:58:42.871930+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5193}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5193': {'gid': 5193, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6833/924398795', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 924398795}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 924398795}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-16T21:58:50.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-16T21:58:50.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-16T21:58:50.761 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1feb308640 1 Processor -- start 2023-12-16T21:58:50.762 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1feb308640 1 -- start start 2023-12-16T21:58:50.762 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1feb308640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1fe406c9d0 0x7f1fe4070460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1feb308640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe4072180 0x7f1fe40709a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1feb308640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe40710e0 con 0x7f1fe4144840 2023-12-16T21:58:50.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1feb308640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe415f010 con 0x7f1fe406c9d0 2023-12-16T21:58:50.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1feb308640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4070ee0 con 0x7f1fe4072180 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1fe98a5640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe4072180 0x7f1fe40709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1fe88a3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1fe406c9d0 0x7f1fe4070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1fe98a5640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe4072180 0x7f1fe40709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:54830/0 (socket says 172.21.15.28:54830) 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1fe88a3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1fe406c9d0 0x7f1fe4070460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:49370/0 (socket says 172.21.15.28:49370) 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1fe98a5640 1 -- 172.21.15.28:0/415203515 learned_addr learned my addr 172.21.15.28:0/415203515 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1fe98a5640 1 -- 172.21.15.28:0/415203515 >> v2:172.21.15.122:3300/0 conn(0x7f1fe406c9d0 msgr2=0x7f1fe4070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1fe98a5640 1 --2- 172.21.15.28:0/415203515 >> v2:172.21.15.122:3300/0 conn(0x7f1fe406c9d0 0x7f1fe4070460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.757+0000 7f1fe98a5640 1 -- 172.21.15.28:0/415203515 >> v1:172.21.15.122:6789/0 conn(0x7f1fe4144840 legacy=0x7f1fe406c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.761+0000 7f1fe98a5640 1 -- 172.21.15.28:0/415203515 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe40713c0 con 0x7f1fe4072180 2023-12-16T21:58:50.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.761+0000 7f1fe98a5640 1 --2- 172.21.15.28:0/415203515 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe4072180 0x7f1fe40709a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009e50 tx=0x7f1fdc01c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6fd14f166e26b33 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.761+0000 7f1fe3fff640 1 -- 172.21.15.28:0/415203515 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0a58a0 con 0x7f1fe4072180 2023-12-16T21:58:50.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.761+0000 7f1fe3fff640 1 -- 172.21.15.28:0/415203515 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc0a5a00 con 0x7f1fe4072180 2023-12-16T21:58:50.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.761+0000 7f1fe3fff640 1 -- 172.21.15.28:0/415203515 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0a5d90 con 0x7f1fe4072180 2023-12-16T21:58:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.761+0000 7f1feb308640 1 -- 172.21.15.28:0/415203515 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe4072180 msgr2=0x7f1fe40709a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.761+0000 7f1feb308640 1 --2- 172.21.15.28:0/415203515 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe4072180 0x7f1fe40709a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009e50 tx=0x7f1fdc01c9e0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- 172.21.15.28:0/415203515 shutdown_connections 2023-12-16T21:58:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 --2- 172.21.15.28:0/415203515 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe4072180 0x7f1fe40709a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 --2- 172.21.15.28:0/415203515 >> v2:172.21.15.122:3300/0 conn(0x7f1fe406c9d0 0x7f1fe4070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- 172.21.15.28:0/415203515 >> 172.21.15.28:0/415203515 conn(0x7f1fe406a530 msgr2=0x7f1fe406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- 172.21.15.28:0/415203515 shutdown_connections 2023-12-16T21:58:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- 172.21.15.28:0/415203515 wait complete. 2023-12-16T21:58:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 Processor -- start 2023-12-16T21:58:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- start start 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe406c9d0 0x7f1fe4139390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4072180 0x7f1fe4159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fe4144840 0x7f1fe413a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4141230 con 0x7f1fe4072180 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4140990 con 0x7f1fe4144840 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4140c70 con 0x7f1fe406c9d0 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4072180 0x7f1fe4159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4072180 0x7f1fe4159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36218/0 (socket says 172.21.15.28:36218) 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 -- 172.21.15.28:0/1368513665 learned_addr learned my addr 172.21.15.28:0/1368513665 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe406c9d0 0x7f1fe4139390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe98a5640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fe4144840 0x7f1fe413a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 -- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe406c9d0 msgr2=0x7f1fe4139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe406c9d0 0x7f1fe4139390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 -- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fe4144840 msgr2=0x7f1fe413a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fe4144840 0x7f1fe413a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 -- 172.21.15.28:0/1368513665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc009b30 con 0x7f1fe4072180 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe406c9d0 0x7f1fe4139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4072180 0x7f1fe4159950 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e900 tx=0x7f1fd800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe1ffb640 1 -- 172.21.15.28:0/1368513665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8009690 con 0x7f1fe4072180 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe1ffb640 1 -- 172.21.15.28:0/1368513665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd800ca00 con 0x7f1fe4072180 2023-12-16T21:58:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- 172.21.15.28:0/1368513665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe4140f50 con 0x7f1fe4072180 2023-12-16T21:58:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1fe1ffb640 1 -- 172.21.15.28:0/1368513665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800cdb0 con 0x7f1fe4072180 2023-12-16T21:58:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.765+0000 7f1feb308640 1 -- 172.21.15.28:0/1368513665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe415a9f0 con 0x7f1fe4072180 2023-12-16T21:58:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe1ffb640 1 -- 172.21.15.28:0/1368513665 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1fd8013470 con 0x7f1fe4072180 2023-12-16T21:58:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe1ffb640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1fd0066470 0x7f1fd0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe1ffb640 1 -- 172.21.15.28:0/1368513665 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fd80841a0 con 0x7f1fe4072180 2023-12-16T21:58:50.774 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1fd0066470 0x7f1fd0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.774 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fee439000 1 Processor -- start 2023-12-16T21:58:50.774 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fee439000 1 -- start start 2023-12-16T21:58:50.774 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1fd0066470 0x7f1fd0068930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4001670 tx=0x7f1fd4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fee439000 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 0x56508c0d9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fee439000 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 0x56508c08f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fee439000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fee439000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508bf56f00 con 0x56508c08f690 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fee439000 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508bee6dd0 con 0x56508c0d9960 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fee439000 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x56508bee4300 con 0x56508c0d9040 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36222/0 (socket says 172.21.15.28:36222) 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 -- 172.21.15.28:0/4126566404 learned_addr learned my addr 172.21.15.28:0/4126566404 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 0x56508c0d9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 0x56508c08f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 -- 172.21.15.28:0/4126566404 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 msgr2=0x56508c0d9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 0x56508c0d9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 -- 172.21.15.28:0/4126566404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 msgr2=0x56508c08f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 0x56508c08f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 -- 172.21.15.28:0/4126566404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508bbd7890 con 0x56508c08f690 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.769+0000 7f1fe98a5640 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08fa70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc01a610 tx=0x7f1fdc004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96397fc8d93f66f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fcaffd640 1 -- 172.21.15.28:0/4126566404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0a5b70 con 0x56508c08f690 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fcaffd640 1 -- 172.21.15.28:0/4126566404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc0a5cd0 con 0x56508c08f690 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fcaffd640 1 -- 172.21.15.28:0/4126566404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0b6620 con 0x56508c08f690 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- 172.21.15.28:0/4126566404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 msgr2=0x56508c08fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08fa70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc01a610 tx=0x7f1fdc004ac0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- 172.21.15.28:0/4126566404 shutdown_connections 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08fa70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 0x56508c08f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 --2- 172.21.15.28:0/4126566404 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 0x56508c0d9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- 172.21.15.28:0/4126566404 >> 172.21.15.28:0/4126566404 conn(0x56508c08b640 msgr2=0x56508c08da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- 172.21.15.28:0/4126566404 shutdown_connections 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- 172.21.15.28:0/4126566404 wait complete. 2023-12-16T21:58:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- start start 2023-12-16T21:58:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 Processor -- start 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08f690 0x56508c09ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c0d9040 0x56508c09b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 0x56508c09b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508bedfbf0 con 0x56508c0d9040 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508be97c60 con 0x56508c0d9960 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x56508be9db00 con 0x56508c08f690 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 10 client.0 _ll_register_callbacks cb 0x56508c07a710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c0d9040 0x56508c09b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c0d9040 0x56508c09b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36224/0 (socket says 172.21.15.28:36224) 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 -- 172.21.15.28:0/4218479083 learned_addr learned my addr 172.21.15.28:0/4218479083 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08f690 0x56508c09ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 -- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08f690 msgr2=0x56508c09ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe98a5640 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 0x56508c09b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08f690 0x56508c09ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 -- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 msgr2=0x56508c09b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 0x56508c09b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 -- 172.21.15.28:0/4218479083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe40713c0 con 0x56508c0d9040 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08f690 0x56508c09ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c0d9040 0x56508c09b170 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800c520 tx=0x7f1fd8055250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265777b2d7899bb9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fc3fff640 1 -- 172.21.15.28:0/4218479083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8059170 con 0x56508c0d9040 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fc3fff640 1 -- 172.21.15.28:0/4218479083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fd805d550 con 0x56508c0d9040 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- 172.21.15.28:0/4218479083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56508be9d490 con 0x56508c0d9040 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fc3fff640 1 -- 172.21.15.28:0/4218479083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd805d0c0 con 0x56508c0d9040 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 10 client.5388 fetch_fsmap learned FSMap version 15 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- 172.21.15.28:0/4218479083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x56508bea0200 con 0x56508c0d9040 2023-12-16T21:58:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fc3fff640 1 -- 172.21.15.28:0/4218479083 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8080020 con 0x56508c0d9040 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fc3fff640 1 -- 172.21.15.28:0/4218479083 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1fd8071d20 con 0x56508c0d9040 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 10 client.5388 fetch_fsmap finished waiting for FSMap version 15 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 10 client.5388 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 20 client.5388 populate_metadata read hostname 'smithi028' 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] conn(0x56508c0e90e0 0x56508c0eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 4 client.5388 mds_command: new command op to 5193 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fee439000 1 -- 172.21.15.28:0/4218479083 --> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56508bede540 con 0x56508c0e90e0 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.773+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] conn(0x56508c0e90e0 0x56508c0eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] conn(0x56508c0e90e0 0x56508c0eb4d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2be5f53f5d34500 server_cookie=bfd2554c9475e136 in_seq=0 out_seq=0 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fc3fff640 10 client.5388 ms_handle_connect on v2:172.21.15.28:6832/924398795 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fc3fff640 1 -- 172.21.15.28:0/4218479083 <== mds.0 v2:172.21.15.28:6832/924398795 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56508bede540 con 0x56508c0e90e0 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fc3fff640 10 client.5388 handle_command_reply: tid=0 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 client.5388 shutdown 2023-12-16T21:58:50.786 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 -- 172.21.15.28:0/4218479083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c0d9040 msgr2=0x56508c09b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c0d9040 0x56508c09b170 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800c520 tx=0x7f1fd8055250 comp rx=0 tx=0).stop 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 -- 172.21.15.28:0/4218479083 shutdown_connections 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9960 0x56508c09b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c0d9040 0x56508c09b170 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] conn(0x56508c0e90e0 0x56508c0eb4d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 --2- 172.21.15.28:0/4218479083 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08f690 0x56508c09ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fc3fff640 0 client.5388 ms_handle_reset on v2:172.21.15.28:6832/924398795 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 -- 172.21.15.28:0/4218479083 >> 172.21.15.28:0/4218479083 conn(0x56508c08b640 msgr2=0x56508c08e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 -- 172.21.15.28:0/4218479083 shutdown_connections 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 1 -- 172.21.15.28:0/4218479083 wait complete. 2023-12-16T21:58:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.777+0000 7f1fee439000 20 client.5388 trim_cache size 0 max 16384 2023-12-16T21:58:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 Processor -- start 2023-12-16T21:58:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- start start 2023-12-16T21:58:50.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08b640 0x56508c08ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9040 0x56508c0eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508c08b500 con 0x56508c08f690 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508bb98940 con 0x56508c0d9040 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x56508bf7a0d0 con 0x56508c08b640 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36230/0 (socket says 172.21.15.28:36230) 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 -- 172.21.15.28:0/1322762854 learned_addr learned my addr 172.21.15.28:0/1322762854 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 -- 172.21.15.28:0/1322762854 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08b640 msgr2=0x56508c08ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1322762854 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08b640 0x56508c08ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 -- 172.21.15.28:0/1322762854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9040 msgr2=0x56508c0eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1322762854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9040 0x56508c0eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 -- 172.21.15.28:0/1322762854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56508be86500 con 0x56508c08f690 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1322762854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08ef80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1fd805d020 tx=0x7f1fd8057000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d896f3795a3cfc47 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fcb7fe640 1 -- 172.21.15.28:0/1322762854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8059170 con 0x56508c08f690 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fcb7fe640 1 -- 172.21.15.28:0/1322762854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8080070 con 0x56508c08f690 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fcb7fe640 1 -- 172.21.15.28:0/1322762854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd805d550 con 0x56508c08f690 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- 172.21.15.28:0/1322762854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 msgr2=0x56508c08ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 --2- 172.21.15.28:0/1322762854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08ef80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1fd805d020 tx=0x7f1fd8057000 comp rx=0 tx=0).stop 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- 172.21.15.28:0/1322762854 shutdown_connections 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 --2- 172.21.15.28:0/1322762854 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c0d9040 0x56508c0eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 --2- 172.21.15.28:0/1322762854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08f690 0x56508c08ef80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 --2- 172.21.15.28:0/1322762854 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c08b640 0x56508c08ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- 172.21.15.28:0/1322762854 >> 172.21.15.28:0/1322762854 conn(0x56508c08b120 msgr2=0x56508c09ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- 172.21.15.28:0/1322762854 shutdown_connections 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- 172.21.15.28:0/1322762854 wait complete. 2023-12-16T21:58:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 -- start start 2023-12-16T21:58:50.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.793+0000 7f1fee439000 1 Processor -- start 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08b640 0x56508c0ed500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c08f690 0x56508c0f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 0x56508c0f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508bbd81b0 con 0x56508c08b640 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56508c0d8a60 con 0x56508c08f690 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x56508c086ac0 con 0x56508c0d9040 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 10 client.0 _ll_register_callbacks cb 0x56508c05af60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08b640 0x56508c0ed500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08b640 0x56508c0ed500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36238/0 (socket says 172.21.15.28:36238) 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 -- 172.21.15.28:0/1993376955 learned_addr learned my addr 172.21.15.28:0/1993376955 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c08f690 0x56508c0f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe98a5640 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 0x56508c0f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 -- 172.21.15.28:0/1993376955 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 msgr2=0x56508c0f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 0x56508c0f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 -- 172.21.15.28:0/1993376955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c08f690 msgr2=0x56508c0f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c08f690 0x56508c0f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 -- 172.21.15.28:0/1993376955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56508bea0200 con 0x56508c08b640 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe90a4640 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08b640 0x56508c0ed500 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x56508bea4200 tx=0x7f1fd4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e039e2a16fb79dac server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 1 -- 172.21.15.28:0/1993376955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40054d0 con 0x56508c08b640 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 1 -- 172.21.15.28:0/1993376955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fd4025640 con 0x56508c08b640 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- 172.21.15.28:0/1993376955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56508c091df0 con 0x56508c08b640 2023-12-16T21:58:50.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 1 -- 172.21.15.28:0/1993376955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd4025800 con 0x56508c08b640 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 10 client.5394 fetch_fsmap learned FSMap version 15 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- 172.21.15.28:0/1993376955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x56508bbd7cc0 con 0x56508c08b640 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 1 -- 172.21.15.28:0/1993376955 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56508c091df0 con 0x56508c08b640 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 1 -- 172.21.15.28:0/1993376955 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1fd4003600 con 0x56508c08b640 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 10 client.5394 fetch_fsmap finished waiting for FSMap version 15 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 10 client.5394 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 20 client.5394 populate_metadata read hostname 'smithi028' 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] conn(0x56508c0d9960 0x56508c139c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 4 client.5394 mds_command: new command op to 5193 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- 172.21.15.28:0/1993376955 --> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56508be66a50 con 0x56508c0d9960 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] conn(0x56508c0d9960 0x56508c139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 10 client.5394 ms_handle_connect on v2:172.21.15.28:6832/924398795 2023-12-16T21:58:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fe88a3640 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] conn(0x56508c0d9960 0x56508c139c10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6a2e71d875ea5bd server_cookie=427f78f9f3c1ef6d in_seq=0 out_seq=0 2023-12-16T21:58:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 1 -- 172.21.15.28:0/1993376955 <== mds.0 v2:172.21.15.28:6832/924398795 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x56508be66a50 con 0x56508c0d9960 2023-12-16T21:58:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 10 client.5394 handle_command_reply: tid=0 2023-12-16T21:58:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 client.5394 shutdown 2023-12-16T21:58:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- 172.21.15.28:0/1993376955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08b640 msgr2=0x56508c0ed500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08b640 0x56508c0ed500 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x56508bea4200 tx=0x7f1fd4000bd0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- 172.21.15.28:0/1993376955 shutdown_connections 2023-12-16T21:58:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.28:6832/924398795,v1:172.21.15.28:6833/924398795] conn(0x56508c0d9960 0x56508c139c10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x56508c0d9040 0x56508c0f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56508c08f690 0x56508c0f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fc3fff640 0 client.5394 ms_handle_reset on v2:172.21.15.28:6832/924398795 2023-12-16T21:58:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 --2- 172.21.15.28:0/1993376955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56508c08b640 0x56508c0ed500 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- 172.21.15.28:0/1993376955 >> 172.21.15.28:0/1993376955 conn(0x56508c08b120 msgr2=0x56508c09b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- 172.21.15.28:0/1993376955 shutdown_connections 2023-12-16T21:58:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 1 -- 172.21.15.28:0/1993376955 wait complete. 2023-12-16T21:58:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1fee439000 20 client.5394 trim_cache size 0 max 16384 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "fa65dfdd-b227-4a96-8047-b713453c357b", 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5193, 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 9.3035257379999994, 2023-12-16T21:58:50.811 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 15, 2023-12-16T21:58:50.812 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 36, 2023-12-16T21:58:50.812 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 36, 2023-12-16T21:58:50.812 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 16.516913209999998 2023-12-16T21:58:50.812 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-16T21:58:50.812 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 -- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1fd0066470 msgr2=0x7f1fd0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1fd0066470 0x7f1fd0068930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4001670 tx=0x7f1fd4009040 comp rx=0 tx=0).stop 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 -- 172.21.15.28:0/1368513665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4072180 msgr2=0x7f1fe4159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4072180 0x7f1fe4159950 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e900 tx=0x7f1fd800edc0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 -- 172.21.15.28:0/1368513665 shutdown_connections 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1fd0066470 0x7f1fd0068930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fe4144840 0x7f1fe413a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4072180 0x7f1fe4159950 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 --2- 172.21.15.28:0/1368513665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1fe406c9d0 0x7f1fe4139390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 -- 172.21.15.28:0/1368513665 >> 172.21.15.28:0/1368513665 conn(0x7f1fe406a530 msgr2=0x7f1fe4143880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 -- 172.21.15.28:0/1368513665 shutdown_connections 2023-12-16T21:58:50.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.797+0000 7f1feb308640 1 -- 172.21.15.28:0/1368513665 wait complete. 2023-12-16T21:58:50.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:58:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2658ea6640 1 Processor -- start 2023-12-16T21:58:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2658ea6640 1 -- start start 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2658ea6640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26540b35b0 0x7f26540d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2658ea6640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540b3c70 0x7f265415f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2658ea6640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f26540cf780 0x7f26540cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2658ea6640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540d4090 con 0x7f26540b35b0 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2658ea6640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540d0130 con 0x7f26540cf780 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2658ea6640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26540d3880 con 0x7f26540b3c70 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f265259c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26540b35b0 0x7f26540d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f265259c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26540b35b0 0x7f26540d0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36248/0 (socket says 172.21.15.28:36248) 2023-12-16T21:58:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f265259c640 1 -- 172.21.15.28:0/3285626723 learned_addr learned my addr 172.21.15.28:0/3285626723 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2651d9b640 1 --2- 172.21.15.28:0/3285626723 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540b3c70 0x7f265415f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f2652d9d640 1 --2- 172.21.15.28:0/3285626723 >> v2:172.21.15.122:3300/0 conn(0x7f26540cf780 0x7f26540cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f265259c640 1 -- 172.21.15.28:0/3285626723 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540b3c70 msgr2=0x7f265415f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f265259c640 1 --2- 172.21.15.28:0/3285626723 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540b3c70 0x7f265415f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f265259c640 1 -- 172.21.15.28:0/3285626723 >> v2:172.21.15.122:3300/0 conn(0x7f26540cf780 msgr2=0x7f26540cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:58:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f265259c640 1 --2- 172.21.15.28:0/3285626723 >> v2:172.21.15.122:3300/0 conn(0x7f26540cf780 0x7f26540cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.925+0000 7f265259c640 1 -- 172.21.15.28:0/3285626723 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540cdc80 con 0x7f26540b35b0 2023-12-16T21:58:50.935 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.929+0000 7f265259c640 1 --2- 172.21.15.28:0/3285626723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b35b0 0x7f26540d0f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f265408c5e0 tx=0x7f264401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c19abcb228ecc5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.929+0000 7f265159a640 1 -- 172.21.15.28:0/3285626723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440aa620 con 0x7f26540b35b0 2023-12-16T21:58:50.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.929+0000 7f265159a640 1 -- 172.21.15.28:0/3285626723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26440aa780 con 0x7f26540b35b0 2023-12-16T21:58:50.936 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.929+0000 7f265159a640 1 -- 172.21.15.28:0/3285626723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440aab10 con 0x7f26540b35b0 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- 172.21.15.28:0/3285626723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b35b0 msgr2=0x7f26540d0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 --2- 172.21.15.28:0/3285626723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b35b0 0x7f26540d0f50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f265408c5e0 tx=0x7f264401c500 comp rx=0 tx=0).stop 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- 172.21.15.28:0/3285626723 shutdown_connections 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 --2- 172.21.15.28:0/3285626723 >> v2:172.21.15.122:3300/0 conn(0x7f26540cf780 0x7f26540cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 --2- 172.21.15.28:0/3285626723 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540b3c70 0x7f265415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 --2- 172.21.15.28:0/3285626723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b35b0 0x7f26540d0f50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- 172.21.15.28:0/3285626723 >> 172.21.15.28:0/3285626723 conn(0x7f265409c4b0 msgr2=0x7f26540d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- 172.21.15.28:0/3285626723 shutdown_connections 2023-12-16T21:58:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- 172.21.15.28:0/3285626723 wait complete. 2023-12-16T21:58:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 Processor -- start 2023-12-16T21:58:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- start start 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26540b35b0 0x7f265415bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b3c70 0x7f265414f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540cf780 0x7f2654158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265414fdf0 con 0x7f26540b3c70 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2654154ba0 con 0x7f26540b35b0 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2654154e40 con 0x7f26540cf780 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b3c70 0x7f265414f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b3c70 0x7f265414f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36264/0 (socket says 172.21.15.28:36264) 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 -- 172.21.15.28:0/2959683921 learned_addr learned my addr 172.21.15.28:0/2959683921 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2652d9d640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540cf780 0x7f2654158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f265259c640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26540b35b0 0x7f265415bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 -- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540cf780 msgr2=0x7f2654158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540cf780 0x7f2654158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 -- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26540b35b0 msgr2=0x7f265415bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26540b35b0 0x7f265415bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 -- 172.21.15.28:0/2959683921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26440094d0 con 0x7f26540b3c70 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f265259c640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26540b35b0 0x7f265415bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2651d9b640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b3c70 0x7f265414f5c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f264800d800 tx=0x7f264800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2642ffd640 1 -- 172.21.15.28:0/2959683921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26480096a0 con 0x7f26540b3c70 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2642ffd640 1 -- 172.21.15.28:0/2959683921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2648009800 con 0x7f26540b3c70 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2642ffd640 1 -- 172.21.15.28:0/2959683921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648009b70 con 0x7f26540b3c70 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2654150040 con 0x7f26540b3c70 2023-12-16T21:58:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.933+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26541589a0 con 0x7f26540b3c70 2023-12-16T21:58:50.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.937+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2654069440 con 0x7f26540b3c70 2023-12-16T21:58:50.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.937+0000 7f2642ffd640 1 -- 172.21.15.28:0/2959683921 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f2648009cd0 con 0x7f26540b3c70 2023-12-16T21:58:50.942 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.937+0000 7f2642ffd640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f26340664a0 0x7f2634068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:58:50.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.937+0000 7f2642ffd640 1 -- 172.21.15.28:0/2959683921 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26480844e0 con 0x7f26540b3c70 2023-12-16T21:58:50.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.937+0000 7f265259c640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f26340664a0 0x7f2634068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:58:50.943 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.937+0000 7f265259c640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f26340664a0 0x7f2634068960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f26440ae000 tx=0x7f26440a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:58:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:50.941+0000 7f2642ffd640 1 -- 172.21.15.28:0/2959683921 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f26480519f0 con 0x7f26540b3c70 2023-12-16T21:58:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.129+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f26540d0720 con 0x7f26540b3c70 2023-12-16T21:58:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2642ffd640 1 -- 172.21.15.28:0/2959683921 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2648055010 con 0x7f26540b3c70 2023-12-16T21:58:51.136 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f26340664a0 msgr2=0x7f2634068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f26340664a0 0x7f2634068960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f26440ae000 tx=0x7f26440a7000 comp rx=0 tx=0).stop 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b3c70 msgr2=0x7f265414f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b3c70 0x7f265414f5c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f264800d800 tx=0x7f264800dcc0 comp rx=0 tx=0).stop 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 shutdown_connections 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f26340664a0 0x7f2634068960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f26540cf780 0x7f2654158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26540b3c70 0x7f265414f5c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 --2- 172.21.15.28:0/2959683921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f26540b35b0 0x7f265415bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 >> 172.21.15.28:0/2959683921 conn(0x7f265409c4b0 msgr2=0x7f26540c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 shutdown_connections 2023-12-16T21:58:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:51.133+0000 7f2658ea6640 1 -- 172.21.15.28:0/2959683921 wait complete. 2023-12-16T21:58:51.153 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:58:51.153 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:58:51.153 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2023-12-16T21:58:51.153 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:58:51.154 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:51.154 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:58:51.154 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:58:51.154 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:58:51.154 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:58:51.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:51.154 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2023-12-16T21:58:51.160 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:58:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:58:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:58:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:51.160 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42395sec preferred_lft 42395sec 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:51.161 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 02:42:f4:d5:13:f2 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2023-12-16T21:58:51.162 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:51.163 INFO:teuthology.orchestra.run.smithi028.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:58:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:58:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2023-12-16T21:58:51.163 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:58:51.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:51.163 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:58:51.212 INFO:teuthology.orchestra.run.smithi028.stdout:1 2023-12-16T21:58:51.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:51.214 DEBUG:teuthology.orchestra.run.smithi028:> route 2023-12-16T21:58:51.262 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2023-12-16T21:58:51.262 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:58:51.262 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-16T21:58:51.262 INFO:teuthology.orchestra.run.smithi028.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-16T21:58:51.262 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-16T21:58:51.262 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:58:51.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:51.263 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2023-12-16T21:58:51.263 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2023-12-16T21:58:51.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:58:51.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:58:51.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:58:51.263 DEBUG:teuthology.orchestra.run.smithi028:> ') 2023-12-16T21:58:51.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:51.338 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2023-12-16T21:58:51.385 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-16T21:58:51.385 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:58:51.385 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:58:51.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:51.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2023-12-16T21:58:51.441 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:58:51.441 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:58:51.441 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:51.489 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:58:51.489 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:58:51.489 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:51.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2023-12-16T21:58:51.599 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:51.650 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2023-12-16T21:58:51.650 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2023-12-16T21:58:51.650 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2023-12-16T21:58:51.650 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2023-12-16T21:58:51.650 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2023-12-16T21:58:51.650 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2023-12-16T21:58:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:58:51.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:51.653 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2023-12-16T21:58:51.696 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:58:51.697 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:58:51.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2023-12-16T21:58:51.756 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:51.804 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2023-12-16T21:58:51.804 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2023-12-16T21:58:51.804 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2023-12-16T21:58:51.804 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2023-12-16T21:58:51.804 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2023-12-16T21:58:51.805 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:58:51.806 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:58:51.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:51.807 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2023-12-16T21:58:51.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:2023-12-16T21:58:51.821+0000 7ff79e9dd580 -1 init, newargv = 0x564dc0999a80 newargc=15 2023-12-16T21:58:51.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[47505]: starting ceph client 2023-12-16T21:58:51.830 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[47505]: starting fuse 2023-12-16T21:58:52.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2023-12-16T21:58:52.947 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:58:52.999 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2023-12-16T21:58:52.999 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2023-12-16T21:58:52.999 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2023-12-16T21:58:52.999 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2023-12-16T21:58:52.999 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2023-12-16T21:58:52.999 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/run/lock 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/fs/nfsd 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2023-12-16T21:58:53.000 INFO:teuthology.orchestra.run.smithi028.stdout:/run/rpc_pipefs 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:58:53.001 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:53.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:53.002 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2023-12-16T21:58:53.049 INFO:teuthology.orchestra.run.smithi028.stdout:45 2023-12-16T21:58:53.049 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-12-16T21:58:53.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> import re 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> import os 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2023-12-16T21:58:53.050 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2023-12-16T21:58:53.051 DEBUG:teuthology.orchestra.run.smithi028:> return f 2023-12-16T21:58:53.052 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:58:53.052 DEBUG:teuthology.orchestra.run.smithi028:> 2023-12-16T21:58:53.052 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.0")) 2023-12-16T21:58:53.052 DEBUG:teuthology.orchestra.run.smithi028:> ' 2023-12-16T21:58:53.181 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.0.47505.asok 2023-12-16T21:58:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:58:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:58:53.193 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47505.asok 2023-12-16T21:58:53.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:53.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47505.asok status 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "0", 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "47505", 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:53.352 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5406, 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 5406 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2970286834 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:53.353 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2970286834 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.5406 192.168.0.1:0/2970286834", 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/2970286834", 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 15, 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 36, 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2023-12-16T21:58:53.354 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-16T21:58:53.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:53.366 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2023-12-16T21:58:53.367 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:53.372 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2023-12-16T21:58:53.373 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:53.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:53.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:53.428 DEBUG:teuthology.orchestra.run.smithi028:> mktemp -d --suffix=xfstests-dev 2023-12-16T21:58:53.480 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.nC0MjoAvrDxfstests-dev 2023-12-16T21:58:53.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:53.480 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo chmod a+rx /tmp/tmp.nC0MjoAvrDxfstests-dev) 2023-12-16T21:58:53.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:53.538 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec git clone https://git.ceph.com/xfstests-dev.git --depth 1 /tmp/tmp.nC0MjoAvrDxfstests-dev) 2023-12-16T21:58:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:Cloning into '/tmp/tmp.nC0MjoAvrDxfstests-dev'... 2023-12-16T21:58:54.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:54.360 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir test) 2023-12-16T21:58:54.414 DEBUG:teuthology.orchestra.run.smithi028:> mktemp -d --suffix=test 2023-12-16T21:58:54.461 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.XR9uc7m4Witest 2023-12-16T21:58:54.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:54.461 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir scratch) 2023-12-16T21:58:54.514 DEBUG:teuthology.orchestra.run.smithi028:> mktemp -d --suffix=scratch 2023-12-16T21:58:54.560 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.0nCImS6pLqscratch 2023-12-16T21:58:54.560 INFO:tasks.cephfs.xfstests_dev:distro and version detected is "ubuntu" and "22.04". 2023-12-16T21:58:54.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo apt-get install -y acl attr automake bc dbench dump e2fsprogs fio gawk gcc git indent libacl1-dev libaio-dev libcap-dev libgdbm-dev libtool libtool-bin liburing-dev libuuid1 lvm2 make psmisc python3 quota sed uuid-dev uuid-runtime xfsprogs sqlite3 exfatprogs f2fs-tools ocfs2-tools udftools xfsdump xfslibs-dev 2023-12-16T21:58:54.675 INFO:teuthology.orchestra.run.smithi028.stdout:Reading package lists... 2023-12-16T21:58:54.875 INFO:teuthology.orchestra.run.smithi028.stdout:Building dependency tree... 2023-12-16T21:58:54.876 INFO:teuthology.orchestra.run.smithi028.stdout:Reading state information... 2023-12-16T21:58:55.055 INFO:teuthology.orchestra.run.smithi028.stdout:acl is already the newest version (2.3.1-1). 2023-12-16T21:58:55.055 INFO:teuthology.orchestra.run.smithi028.stdout:acl set to manually installed. 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:attr is already the newest version (1:2.5.1-1build1). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:automake is already the newest version (1:1.16.5-1.3). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:bc is already the newest version (1.07.1-3build1). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:gcc is already the newest version (4:11.2.0-1ubuntu1). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:gcc set to manually installed. 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:libacl1-dev is already the newest version (2.3.1-1). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:libaio-dev is already the newest version (0.3.112-13build1). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:libtool is already the newest version (2.4.6-15build2). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:libuuid1 is already the newest version (2.37.2-4ubuntu3). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:lvm2 is already the newest version (2.03.11-2.1ubuntu4). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:make is already the newest version (4.3-4.1build1). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:make set to manually installed. 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:psmisc is already the newest version (23.4-2build3). 2023-12-16T21:58:55.056 INFO:teuthology.orchestra.run.smithi028.stdout:quota is already the newest version (4.06-1build2). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:sed is already the newest version (4.8-1ubuntu2). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:uuid-runtime is already the newest version (2.37.2-4ubuntu3). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:xfsdump is already the newest version (3.1.9+0+nmu2). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:xfsprogs is already the newest version (5.13.0-1ubuntu2). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:dbench is already the newest version (4.0-2build1). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:dump is already the newest version (0.4b47-3). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:e2fsprogs is already the newest version (1.46.5-2ubuntu1.1). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:git is already the newest version (1:2.34.1-1ubuntu1.10). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:indent is already the newest version (2.2.12-1ubuntu0.22.04.1). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:python3 is already the newest version (3.10.6-1~22.04). 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages were automatically installed and are no longer required: 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-12-16T21:58:55.057 INFO:teuthology.orchestra.run.smithi028.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-12-16T21:58:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-12-16T21:58:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-12-16T21:58:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: python2.7 python2.7-dev python2.7-minimal 2023-12-16T21:58:55.058 INFO:teuthology.orchestra.run.smithi028.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-16T21:58:55.058 INFO:teuthology.orchestra.run.smithi028.stdout:The following additional packages will be installed: 2023-12-16T21:58:55.058 INFO:teuthology.orchestra.run.smithi028.stdout: libcap2 libcmap4 libcorosync-common4 libdlm3 libpmemblk1 libqb100 2023-12-16T21:58:55.059 INFO:teuthology.orchestra.run.smithi028.stdout:Suggested packages: 2023-12-16T21:58:55.059 INFO:teuthology.orchestra.run.smithi028.stdout: gnuplot gfio python-scipy gawk-doc sqlite3-doc dvd+rw-tools udfclient 2023-12-16T21:58:55.093 INFO:teuthology.orchestra.run.smithi028.stdout:The following NEW packages will be installed: 2023-12-16T21:58:55.093 INFO:teuthology.orchestra.run.smithi028.stdout: exfatprogs f2fs-tools fio libcap-dev libcmap4 libcorosync-common4 libdlm3 2023-12-16T21:58:55.094 INFO:teuthology.orchestra.run.smithi028.stdout: libgdbm-dev libpmemblk1 libqb100 liburing-dev ocfs2-tools sqlite3 udftools 2023-12-16T21:58:55.095 INFO:teuthology.orchestra.run.smithi028.stdout:The following packages will be upgraded: 2023-12-16T21:58:55.096 INFO:teuthology.orchestra.run.smithi028.stdout: gawk libcap2 2023-12-16T21:58:56.163 INFO:teuthology.orchestra.run.smithi028.stdout:2 upgraded, 14 newly installed, 0 to remove and 262 not upgraded. 2023-12-16T21:58:56.163 INFO:teuthology.orchestra.run.smithi028.stdout:Need to get 6,641 kB of archives. 2023-12-16T21:58:56.163 INFO:teuthology.orchestra.run.smithi028.stdout:After this operation, 14.2 MB of additional disk space will be used. 2023-12-16T21:58:56.163 INFO:teuthology.orchestra.run.smithi028.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gawk amd64 1:5.1.0-1ubuntu0.1 [447 kB] 2023-12-16T21:58:56.692 INFO:teuthology.orchestra.run.smithi028.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libcorosync-common4 amd64 3.1.6-1ubuntu1 [7,322 B] 2023-12-16T21:58:56.693 INFO:teuthology.orchestra.run.smithi028.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqb100 amd64 2.0.4-1ubuntu0.2 [74.2 kB] 2023-12-16T21:58:56.703 INFO:teuthology.orchestra.run.smithi028.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 libcmap4 amd64 3.1.6-1ubuntu1 [10.4 kB] 2023-12-16T21:58:56.704 INFO:teuthology.orchestra.run.smithi028.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libdlm3 amd64 4.1.1-1build1 [14.4 kB] 2023-12-16T21:58:56.706 INFO:teuthology.orchestra.run.smithi028.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 ocfs2-tools amd64 1.8.7-1build1 [646 kB] 2023-12-16T21:58:56.812 INFO:teuthology.orchestra.run.smithi028.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libcap2 amd64 1:2.44-1ubuntu0.22.04.1 [18.3 kB] 2023-12-16T21:58:56.814 INFO:teuthology.orchestra.run.smithi028.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 exfatprogs amd64 1.1.3-1 [40.3 kB] 2023-12-16T21:58:56.819 INFO:teuthology.orchestra.run.smithi028.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 f2fs-tools amd64 1.14.0-2build1 [169 kB] 2023-12-16T21:58:56.838 INFO:teuthology.orchestra.run.smithi028.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-12-16T21:58:56.845 INFO:teuthology.orchestra.run.smithi028.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-12-16T21:58:57.180 INFO:teuthology.orchestra.run.smithi028.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libcap-dev amd64 1:2.44-1ubuntu0.22.04.1 [39.4 kB] 2023-12-16T21:58:57.427 INFO:teuthology.orchestra.run.smithi028.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libgdbm-dev amd64 1.23-1 [117 kB] 2023-12-16T21:58:57.577 INFO:teuthology.orchestra.run.smithi028.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liburing-dev amd64 2.1-2build1 [51.8 kB] 2023-12-16T21:58:57.614 INFO:teuthology.orchestra.run.smithi028.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 sqlite3 amd64 3.37.2-2ubuntu0.1 [768 kB] 2023-12-16T21:58:57.811 INFO:teuthology.orchestra.run.smithi028.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 udftools amd64 2.3-1 [125 kB] 2023-12-16T21:58:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:debconf: unable to initialize frontend: Dialog 2023-12-16T21:58:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2023-12-16T21:58:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:debconf: falling back to frontend: Readline 2023-12-16T21:58:58.158 INFO:teuthology.orchestra.run.smithi028.stderr:debconf: unable to initialize frontend: Readline 2023-12-16T21:58:58.158 INFO:teuthology.orchestra.run.smithi028.stderr:debconf: (This frontend requires a controlling tty.) 2023-12-16T21:58:58.158 INFO:teuthology.orchestra.run.smithi028.stderr:debconf: falling back to frontend: Teletype 2023-12-16T21:58:58.162 INFO:teuthology.orchestra.run.smithi028.stderr:dpkg-preconfigure: unable to re-open stdin: 2023-12-16T21:58:58.193 INFO:teuthology.orchestra.run.smithi028.stdout:Fetched 6,641 kB in 3s (2,441 kB/s) 2023-12-16T21:58:58.475 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176425 files and directories currently installed.) 2023-12-16T21:58:58.482 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../0-gawk_1%3a5.1.0-1ubuntu0.1_amd64.deb ... 2023-12-16T21:58:58.615 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking gawk (1:5.1.0-1ubuntu0.1) over (1:5.1.0-1build3) ... 2023-12-16T21:58:59.142 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcorosync-common4:amd64. 2023-12-16T21:58:59.172 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../1-libcorosync-common4_3.1.6-1ubuntu1_amd64.deb ... 2023-12-16T21:58:59.251 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcorosync-common4:amd64 (3.1.6-1ubuntu1) ... 2023-12-16T21:58:59.696 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libqb100:amd64. 2023-12-16T21:58:59.725 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../2-libqb100_2.0.4-1ubuntu0.2_amd64.deb ... 2023-12-16T21:58:59.763 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libqb100:amd64 (2.0.4-1ubuntu0.2) ... 2023-12-16T21:59:00.041 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcmap4:amd64. 2023-12-16T21:59:00.057 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../3-libcmap4_3.1.6-1ubuntu1_amd64.deb ... 2023-12-16T21:59:00.090 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcmap4:amd64 (3.1.6-1ubuntu1) ... 2023-12-16T21:59:00.452 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libdlm3:amd64. 2023-12-16T21:59:00.482 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../4-libdlm3_4.1.1-1build1_amd64.deb ... 2023-12-16T21:59:00.518 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libdlm3:amd64 (4.1.1-1build1) ... 2023-12-16T21:59:00.857 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package ocfs2-tools. 2023-12-16T21:59:00.885 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../5-ocfs2-tools_1.8.7-1build1_amd64.deb ... 2023-12-16T21:59:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking ocfs2-tools (1.8.7-1build1) ... 2023-12-16T21:59:01.357 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../6-libcap2_1%3a2.44-1ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:59:01.486 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcap2:amd64 (1:2.44-1ubuntu0.22.04.1) over (1:2.44-1build3) ... 2023-12-16T21:59:01.841 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcap2:amd64 (1:2.44-1ubuntu0.22.04.1) ... 2023-12-16T21:59:02.145 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package exfatprogs. 2023-12-16T21:59:02.233 INFO:teuthology.orchestra.run.smithi028.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176499 files and directories currently installed.) 2023-12-16T21:59:02.239 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../0-exfatprogs_1.1.3-1_amd64.deb ... 2023-12-16T21:59:02.285 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking exfatprogs (1.1.3-1) ... 2023-12-16T21:59:02.564 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package f2fs-tools. 2023-12-16T21:59:02.581 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../1-f2fs-tools_1.14.0-2build1_amd64.deb ... 2023-12-16T21:59:02.665 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking f2fs-tools (1.14.0-2build1) ... 2023-12-16T21:59:03.002 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-12-16T21:59:03.031 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../2-libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-12-16T21:59:03.068 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-12-16T21:59:03.405 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package fio. 2023-12-16T21:59:03.434 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../3-fio_3.28-1_amd64.deb ... 2023-12-16T21:59:03.473 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking fio (3.28-1) ... 2023-12-16T21:59:03.927 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libcap-dev:amd64. 2023-12-16T21:59:03.956 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../4-libcap-dev_1%3a2.44-1ubuntu0.22.04.1_amd64.deb ... 2023-12-16T21:59:03.994 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libcap-dev:amd64 (1:2.44-1ubuntu0.22.04.1) ... 2023-12-16T21:59:04.249 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package libgdbm-dev:amd64. 2023-12-16T21:59:04.278 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../5-libgdbm-dev_1.23-1_amd64.deb ... 2023-12-16T21:59:04.324 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking libgdbm-dev:amd64 (1.23-1) ... 2023-12-16T21:59:04.660 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package liburing-dev:amd64. 2023-12-16T21:59:04.678 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../6-liburing-dev_2.1-2build1_amd64.deb ... 2023-12-16T21:59:04.719 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking liburing-dev:amd64 (2.1-2build1) ... 2023-12-16T21:59:05.022 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package sqlite3. 2023-12-16T21:59:05.051 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../7-sqlite3_3.37.2-2ubuntu0.1_amd64.deb ... 2023-12-16T21:59:05.088 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking sqlite3 (3.37.2-2ubuntu0.1) ... 2023-12-16T21:59:05.475 INFO:teuthology.orchestra.run.smithi028.stdout:Selecting previously unselected package udftools. 2023-12-16T21:59:05.504 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing to unpack .../8-udftools_2.3-1_amd64.deb ... 2023-12-16T21:59:05.554 INFO:teuthology.orchestra.run.smithi028.stdout:Unpacking udftools (2.3-1) ... 2023-12-16T21:59:05.869 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up exfatprogs (1.1.3-1) ... 2023-12-16T21:59:05.996 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up gawk (1:5.1.0-1ubuntu0.1) ... 2023-12-16T21:59:06.206 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up f2fs-tools (1.14.0-2build1) ... 2023-12-16T21:59:06.333 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libqb100:amd64 (2.0.4-1ubuntu0.2) ... 2023-12-16T21:59:06.459 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcorosync-common4:amd64 (3.1.6-1ubuntu1) ... 2023-12-16T21:59:06.585 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-12-16T21:59:06.711 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up udftools (2.3-1) ... 2023-12-16T21:59:06.862 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libdlm3:amd64 (4.1.1-1build1) ... 2023-12-16T21:59:06.988 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libgdbm-dev:amd64 (1.23-1) ... 2023-12-16T21:59:07.114 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcap-dev:amd64 (1:2.44-1ubuntu0.22.04.1) ... 2023-12-16T21:59:07.215 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up sqlite3 (3.37.2-2ubuntu0.1) ... 2023-12-16T21:59:07.316 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up liburing-dev:amd64 (2.1-2build1) ... 2023-12-16T21:59:07.433 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up libcmap4:amd64 (3.1.6-1ubuntu1) ... 2023-12-16T21:59:07.559 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up fio (3.28-1) ... 2023-12-16T21:59:08.052 INFO:teuthology.orchestra.run.smithi028.stdout:Setting up ocfs2-tools (1.8.7-1build1) ... 2023-12-16T21:59:08.324 INFO:teuthology.orchestra.run.smithi028.stdout:debconf: unable to initialize frontend: Dialog 2023-12-16T21:59:08.324 INFO:teuthology.orchestra.run.smithi028.stdout:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2023-12-16T21:59:08.324 INFO:teuthology.orchestra.run.smithi028.stdout:debconf: falling back to frontend: Readline 2023-12-16T21:59:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for initramfs-tools (0.140ubuntu13.1) ... 2023-12-16T21:59:09.289 INFO:teuthology.orchestra.run.smithi028.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2023-12-16T21:59:13.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:59:13.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:59:27.675 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-16T21:59:27.827 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-16T21:59:29.171 INFO:teuthology.orchestra.run.smithi028.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-16T21:59:29.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo useradd -m fsgqa 2023-12-16T21:59:29.699 INFO:teuthology.orchestra.run.smithi028.stderr:useradd: user 'fsgqa' already exists 2023-12-16T21:59:29.700 DEBUG:teuthology.orchestra.run:got remote process result: 9 2023-12-16T21:59:29.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo groupadd fsgqa 2023-12-16T21:59:29.754 INFO:teuthology.orchestra.run.smithi028.stderr:groupadd: group 'fsgqa' already exists 2023-12-16T21:59:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 9 2023-12-16T21:59:29.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo useradd fsgqa2 2023-12-16T21:59:30.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo useradd 123456-fsgqa 2023-12-16T21:59:30.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json mon dump 2023-12-16T21:59:30.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1de78a640 1 Processor -- start 2023-12-16T21:59:30.730 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1de78a640 1 -- start start 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1de78a640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fe1d8144840 0x7fe1d806c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1de78a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8072180 0x7fe1d806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1de78a640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d80710e0 con 0x7fe1d8070460 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1de78a640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d815f010 con 0x7fe1d8144840 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1de78a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80713c0 con 0x7fe1d8072180 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1dcd27640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1d8070460 0x7fe1d80708e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48816/0 (socket says 172.21.15.28:48816) 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1dcd27640 1 -- 172.21.15.28:0/1631327218 learned_addr learned my addr 172.21.15.28:0/1631327218 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1cf7fe640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8072180 0x7fe1d806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1d7fff640 1 --2- 172.21.15.28:0/1631327218 >> v2:172.21.15.122:3300/0 conn(0x7fe1d8144840 0x7fe1d806c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096968997 0 0) 0x7fe1d80710e0 con 0x7fe1d8070460 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.727+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c4003530 con 0x7fe1d8070460 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 716345872 0 0) 0x7fe1c4003530 con 0x7fe1d8070460 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8072180 msgr2=0x7fe1d806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8072180 0x7fe1d806c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 >> v2:172.21.15.122:3300/0 conn(0x7fe1d8144840 msgr2=0x7fe1d806c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 --2- 172.21.15.28:0/1631327218 >> v2:172.21.15.122:3300/0 conn(0x7fe1d8144840 0x7fe1d806c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d80716a0 con 0x7fe1d8070460 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fe1d0001a00 con 0x7fe1d8070460 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 >> v1:172.21.15.28:6789/0 conn(0x7fe1d8070460 legacy=0x7fe1d80708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c4003980 0x7fe1c4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8072180 0x7fe1c40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1c4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80713c0 con 0x7fe1c4004d10 2023-12-16T21:59:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c400b990 con 0x7fe1c4003980 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c400bad0 con 0x7fe1d8072180 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1dcd27640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1c4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1cf7fe640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8072180 0x7fe1c40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d7fff640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c4003980 0x7fe1c4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1dcd27640 1 -- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8072180 msgr2=0x7fe1c40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1dcd27640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8072180 0x7fe1c40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1dcd27640 1 -- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c4003980 msgr2=0x7fe1c4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1dcd27640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1c4003980 0x7fe1c4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1dcd27640 1 -- 172.21.15.28:0/1631327218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c4003530 con 0x7fe1c4004d10 2023-12-16T21:59:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1dcd27640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1c4005190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe1d806e770 tx=0x7fe1d001d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c60400894a14b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:59:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00ad630 con 0x7fe1c4004d10 2023-12-16T21:59:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d00a8070 con 0x7fe1c4004d10 2023-12-16T21:59:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1631327218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d00ad9d0 con 0x7fe1c4004d10 2023-12-16T21:59:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1de78a640 1 -- 172.21.15.28:0/1631327218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 msgr2=0x7fe1c4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1de78a640 1 --2- 172.21.15.28:0/1631327218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1c4005190 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe1d806e770 tx=0x7fe1d001d3a0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1dcd27640 1 -- 172.21.15.28:0/1631327218 reap_dead start 2023-12-16T21:59:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1de78a640 1 -- 172.21.15.28:0/1631327218 shutdown_connections 2023-12-16T21:59:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1de78a640 1 -- 172.21.15.28:0/1631327218 >> 172.21.15.28:0/1631327218 conn(0x7fe1d806a530 msgr2=0x7fe1d806a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:59:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1de78a640 1 -- 172.21.15.28:0/1631327218 shutdown_connections 2023-12-16T21:59:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.731+0000 7fe1de78a640 1 -- 172.21.15.28:0/1631327218 wait complete. 2023-12-16T21:59:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 Processor -- start 2023-12-16T21:59:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 -- start start 2023-12-16T21:59:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1d81591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d813e610 0x7fe1d8159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8159d50 0x7fe1d813f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d813e180 con 0x7fe1c4004d10 2023-12-16T21:59:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d813d570 con 0x7fe1d813e610 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d813d850 con 0x7fe1d8159d50 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1d81591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1d81591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43244/0 (socket says 172.21.15.28:43244) 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 -- 172.21.15.28:0/4070772678 learned_addr learned my addr 172.21.15.28:0/4070772678 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1dcd27640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8159d50 0x7fe1d813f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1cf7fe640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d813e610 0x7fe1d8159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 -- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8159d50 msgr2=0x7fe1d813f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8159d50 0x7fe1d813f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 -- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d813e610 msgr2=0x7fe1d8159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d813e610 0x7fe1d8159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 -- 172.21.15.28:0/4070772678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d813db30 con 0x7fe1c4004d10 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1cf7fe640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d813e610 0x7fe1d8159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d7fff640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1d81591a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80115e0 tx=0x7fe1c8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:59:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d57fa640 1 -- 172.21.15.28:0/4070772678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c800f8f0 con 0x7fe1c4004d10 2023-12-16T21:59:30.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d57fa640 1 -- 172.21.15.28:0/4070772678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c800fa50 con 0x7fe1c4004d10 2023-12-16T21:59:30.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d813de10 con 0x7fe1c4004d10 2023-12-16T21:59:30.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1d57fa640 1 -- 172.21.15.28:0/4070772678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c800fdc0 con 0x7fe1c4004d10 2023-12-16T21:59:30.741 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.735+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d813fe00 con 0x7fe1c4004d10 2023-12-16T21:59:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.739+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d8069440 con 0x7fe1c4004d10 2023-12-16T21:59:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.739+0000 7fe1d57fa640 1 -- 172.21.15.28:0/4070772678 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fe1c8015460 con 0x7fe1c4004d10 2023-12-16T21:59:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.739+0000 7fe1d57fa640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe1b8066470 0x7fe1b8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.739+0000 7fe1d57fa640 1 -- 172.21.15.28:0/4070772678 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1c80847a0 con 0x7fe1c4004d10 2023-12-16T21:59:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.739+0000 7fe1cf7fe640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe1b8066470 0x7fe1b8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.739+0000 7fe1cf7fe640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe1b8066470 0x7fe1b8068930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe1d813ba60 tx=0x7fe1c0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:59:30.745 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.743+0000 7fe1d57fa640 1 -- 172.21.15.28:0/4070772678 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fe1c8051c10 con 0x7fe1c4004d10 2023-12-16T21:59:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.959+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1d80c3f30 con 0x7fe1c4004d10 2023-12-16T21:59:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.959+0000 7fe1d57fa640 1 -- 172.21.15.28:0/4070772678 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+1276 (secure 0 0 0) 0x7fe1c8055230 con 0x7fe1c4004d10 2023-12-16T21:59:30.963 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T21:59:30.963 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","modified":"2023-12-16T21:56:45.365130Z","created":"2023-12-16T21:56:45.365130Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3301","nonce":0},{"type":"v1","addr":"172.21.15.122:6790","nonce":0}]},"addr":"172.21.15.122:6790/0","public_addr":"172.21.15.122:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-12-16T21:59:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe1b8066470 msgr2=0x7fe1b8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe1b8066470 0x7fe1b8068930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe1d813ba60 tx=0x7fe1c0008040 comp rx=0 tx=0).stop 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 msgr2=0x7fe1d81591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1d81591a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80115e0 tx=0x7fe1c8011aa0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 shutdown_connections 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fe1b8066470 0x7fe1b8068930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fe1d8159d50 0x7fe1d813f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d813e610 0x7fe1d8159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 --2- 172.21.15.28:0/4070772678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1c4004d10 0x7fe1d81591a0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 >> 172.21.15.28:0/4070772678 conn(0x7fe1d806a530 msgr2=0x7fe1d80cc130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:59:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 shutdown_connections 2023-12-16T21:59:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:30.963+0000 7fe1de78a640 1 -- 172.21.15.28:0/4070772678 wait complete. 2023-12-16T21:59:30.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth get client.admin 2023-12-16T21:59:31.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f72a064e640 1 Processor -- start 2023-12-16T21:59:31.097 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f72a064e640 1 -- start start 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f72a064e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980b35b0 0x7f72980d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f72a064e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f72980cfe90 0x7f729815f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f72a064e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72980d3880 con 0x7f72980b3c70 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f72a064e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72980d4090 con 0x7f72980cfe90 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f72a064e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72980cdc80 con 0x7f72980b35b0 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f729ebeb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f72980cfe90 0x7f729815f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f729ebeb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f72980cfe90 0x7f729815f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43250/0 (socket says 172.21.15.28:43250) 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f729ebeb640 1 -- 172.21.15.28:0/3170445007 learned_addr learned my addr 172.21.15.28:0/3170445007 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f729e3ea640 1 --2- 172.21.15.28:0/3170445007 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980b35b0 0x7f72980d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f729ebeb640 1 -- 172.21.15.28:0/3170445007 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980b35b0 msgr2=0x7f72980d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f729ebeb640 1 --2- 172.21.15.28:0/3170445007 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980b35b0 0x7f72980d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f729ebeb640 1 -- 172.21.15.28:0/3170445007 >> v1:172.21.15.122:6789/0 conn(0x7f72980b3c70 legacy=0x7f72980cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:59:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.095+0000 7f729ebeb640 1 -- 172.21.15.28:0/3170445007 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980d3ad0 con 0x7f72980cfe90 2023-12-16T21:59:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f729ebeb640 1 --2- 172.21.15.28:0/3170445007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980cfe90 0x7f729815f060 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7290009e50 tx=0x7f729001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42365a17f3d15bce server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:59:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f729d3e8640 1 -- 172.21.15.28:0/3170445007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72900a5850 con 0x7f72980cfe90 2023-12-16T21:59:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f729d3e8640 1 -- 172.21.15.28:0/3170445007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72900a59b0 con 0x7f72980cfe90 2023-12-16T21:59:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f729d3e8640 1 -- 172.21.15.28:0/3170445007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72900a5d40 con 0x7f72980cfe90 2023-12-16T21:59:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- 172.21.15.28:0/3170445007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980cfe90 msgr2=0x7f729815f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 --2- 172.21.15.28:0/3170445007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980cfe90 0x7f729815f060 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7290009e50 tx=0x7f729001c9d0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- 172.21.15.28:0/3170445007 shutdown_connections 2023-12-16T21:59:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 --2- 172.21.15.28:0/3170445007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980cfe90 0x7f729815f060 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 --2- 172.21.15.28:0/3170445007 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980b35b0 0x7f72980d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- 172.21.15.28:0/3170445007 >> 172.21.15.28:0/3170445007 conn(0x7f729809c4b0 msgr2=0x7f72980d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:59:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- 172.21.15.28:0/3170445007 shutdown_connections 2023-12-16T21:59:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- 172.21.15.28:0/3170445007 wait complete. 2023-12-16T21:59:31.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 Processor -- start 2023-12-16T21:59:31.103 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- start start 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72980b35b0 0x7f729815c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980b3c70 0x7f729815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980cfe90 0x7f729815da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72981532d0 con 0x7f72980b3c70 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7298153760 con 0x7f72980b35b0 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.099+0000 7f72a064e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72980cd2e0 con 0x7f72980cfe90 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980b3c70 0x7f729815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980b3c70 0x7f729815a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43262/0 (socket says 172.21.15.28:43262) 2023-12-16T21:59:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 -- 172.21.15.28:0/4280897178 learned_addr learned my addr 172.21.15.28:0/4280897178 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729ebeb640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980cfe90 0x7f729815da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729e3ea640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72980b35b0 0x7f729815c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 -- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980cfe90 msgr2=0x7f729815da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980cfe90 0x7f729815da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 -- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72980b35b0 msgr2=0x7f729815c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72980b35b0 0x7f729815c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 -- 172.21.15.28:0/4280897178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290009b30 con 0x7f72980b3c70 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729e3ea640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72980b35b0 0x7f729815c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729dbe9640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980b3c70 0x7f729815a140 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f728800c480 tx=0x7f728800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f728effd640 1 -- 172.21.15.28:0/4280897178 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288012830 con 0x7f72980b3c70 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f728effd640 1 -- 172.21.15.28:0/4280897178 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72880129b0 con 0x7f72980b3c70 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72980cd580 con 0x7f72980b3c70 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f728effd640 1 -- 172.21.15.28:0/4280897178 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288012d60 con 0x7f72980b3c70 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7298152f60 con 0x7f72980b3c70 2023-12-16T21:59:31.106 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7298153480 con 0x7f72980b3c70 2023-12-16T21:59:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f728effd640 1 -- 172.21.15.28:0/4280897178 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f728801c460 con 0x7f72980b3c70 2023-12-16T21:59:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f728effd640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f727c0664c0 0x7f727c068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:59:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f728effd640 1 -- 172.21.15.28:0/4280897178 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7288083c00 con 0x7f72980b3c70 2023-12-16T21:59:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729e3ea640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f727c0664c0 0x7f727c068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:59:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.103+0000 7f729e3ea640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f727c0664c0 0x7f727c068980 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7294009840 tx=0x7f7294008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:59:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.107+0000 7f728effd640 1 -- 172.21.15.28:0/4280897178 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7298153480 con 0x7f72980b3c70 2023-12-16T21:59:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth get", "entity": "client.admin"} v 0) v1 -- 0x7f7298153480 con 0x7f72980b3c70 2023-12-16T21:59:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f728effd640 1 -- 172.21.15.28:0/4280897178 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth get", "entity": "client.admin"}]=0 v15) v1 ==== 82+0+151 (secure 0 0 0) 0x7f7298153480 con 0x7f72980b3c70 2023-12-16T21:59:31.314 INFO:teuthology.orchestra.run.smithi028.stdout:[client.admin] 2023-12-16T21:59:31.314 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAdHX5lX91pIRAARh7B+7emWJR5gGjk5UAYTw== 2023-12-16T21:59:31.314 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow *" 2023-12-16T21:59:31.314 INFO:teuthology.orchestra.run.smithi028.stdout: caps mgr = "allow *" 2023-12-16T21:59:31.314 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow *" 2023-12-16T21:59:31.314 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow *" 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f727c0664c0 msgr2=0x7f727c068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f727c0664c0 0x7f727c068980 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7294009840 tx=0x7f7294008040 comp rx=0 tx=0).stop 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980b3c70 msgr2=0x7f729815a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980b3c70 0x7f729815a140 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f728800c480 tx=0x7f728800c940 comp rx=0 tx=0).stop 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 shutdown_connections 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f727c0664c0 0x7f727c068980 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f72980cfe90 0x7f729815da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980b3c70 0x7f729815a140 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 --2- 172.21.15.28:0/4280897178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72980b35b0 0x7f729815c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:59:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.311+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 >> 172.21.15.28:0/4280897178 conn(0x7f729809c4b0 msgr2=0x7f72980b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:59:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.315+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 shutdown_connections 2023-12-16T21:59:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-16T21:59:31.315+0000 7f72a064e640 1 -- 172.21.15.28:0/4280897178 wait complete. 2023-12-16T21:59:31.332 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2023-12-16T21:59:31.337 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.rKan10HCBj 2023-12-16T21:59:31.338 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:59:31.338 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/tmp/tmp.rKan10HCBj 2023-12-16T21:59:31.384 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:59:31.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/tmp/tmp.nC0MjoAvrDxfstests-dev/local.config 2023-12-16T21:59:31.440 INFO:tasks.cephfs.xfstests_dev:local.config's contents - export FSTYP=ceph-fuse export CEPH_FUSE_BIN_PATH=ceph-fuse export TEST_DEV=ceph-fuse # without this tests won't get started export TEST_DIR=/tmp/tmp.XR9uc7m4Witest export SCRATCH_DEV= export SCRATCH_MNT=/tmp/tmp.0nCImS6pLqscratch export CEPHFS_MOUNT_OPTIONS="-m 172.21.15.28:6789 -k /tmp/tmp.rKan10HCBj --client_mountpoint /test" 2023-12-16T21:59:31.440 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-16T21:59:31.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/tmp/tmp.nC0MjoAvrDxfstests-dev/ceph.exclude 2023-12-16T21:59:31.495 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo make) 2023-12-16T21:59:31.551 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize -cfi 2023-12-16T21:59:31.832 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: putting auxiliary files in '.'. 2023-12-16T21:59:31.832 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file './config.guess' 2023-12-16T21:59:31.837 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file './config.sub' 2023-12-16T21:59:31.845 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file './install-sh' 2023-12-16T21:59:31.852 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file './ltmain.sh' 2023-12-16T21:59:31.874 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2023-12-16T21:59:31.874 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file 'm4/libtool.m4' 2023-12-16T21:59:31.880 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2023-12-16T21:59:31.885 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2023-12-16T21:59:31.890 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file 'm4/ltversion.m4' 2023-12-16T21:59:31.896 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2023-12-16T21:59:31.901 INFO:teuthology.orchestra.run.smithi028.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2023-12-16T21:59:31.903 INFO:teuthology.orchestra.run.smithi028.stdout:cp include/install-sh . 2023-12-16T21:59:31.904 INFO:teuthology.orchestra.run.smithi028.stdout:aclocal -I m4 2023-12-16T21:59:33.060 INFO:teuthology.orchestra.run.smithi028.stdout:autoheader 2023-12-16T21:59:33.440 INFO:teuthology.orchestra.run.smithi028.stdout:autoconf 2023-12-16T21:59:33.885 INFO:teuthology.orchestra.run.smithi028.stdout:./configure \ 2023-12-16T21:59:33.885 INFO:teuthology.orchestra.run.smithi028.stdout: --libexecdir=/usr/lib \ 2023-12-16T21:59:33.885 INFO:teuthology.orchestra.run.smithi028.stdout: --exec_prefix=/var/lib 2023-12-16T21:59:34.316 INFO:teuthology.orchestra.run.smithi028.stdout:checking build system type... x86_64-pc-linux-gnu 2023-12-16T21:59:34.316 INFO:teuthology.orchestra.run.smithi028.stdout:checking host system type... x86_64-pc-linux-gnu 2023-12-16T21:59:34.356 INFO:teuthology.orchestra.run.smithi028.stdout:checking how to print strings... printf 2023-12-16T21:59:34.356 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gcc... gcc 2023-12-16T21:59:34.538 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether the C compiler works... yes 2023-12-16T21:59:34.538 INFO:teuthology.orchestra.run.smithi028.stdout:checking for C compiler default output file name... a.out 2023-12-16T21:59:34.573 INFO:teuthology.orchestra.run.smithi028.stdout:checking for suffix of executables... 2023-12-16T21:59:34.665 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether we are cross compiling... no 2023-12-16T21:59:34.685 INFO:teuthology.orchestra.run.smithi028.stdout:checking for suffix of object files... o 2023-12-16T21:59:34.706 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether the compiler supports GNU C... yes 2023-12-16T21:59:34.725 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether gcc accepts -g... yes 2023-12-16T21:59:34.887 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gcc option to enable C11 features... none needed 2023-12-16T21:59:34.893 INFO:teuthology.orchestra.run.smithi028.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2023-12-16T21:59:34.897 INFO:teuthology.orchestra.run.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-12-16T21:59:34.899 INFO:teuthology.orchestra.run.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2023-12-16T21:59:34.901 INFO:teuthology.orchestra.run.smithi028.stdout:checking for fgrep... /usr/bin/grep -F 2023-12-16T21:59:34.908 INFO:teuthology.orchestra.run.smithi028.stdout:checking for ld used by gcc... /usr/bin/ld 2023-12-16T21:59:34.910 INFO:teuthology.orchestra.run.smithi028.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2023-12-16T21:59:34.934 INFO:teuthology.orchestra.run.smithi028.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2023-12-16T21:59:34.957 INFO:teuthology.orchestra.run.smithi028.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2023-12-16T21:59:34.957 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether ln -s works... yes 2023-12-16T21:59:34.972 INFO:teuthology.orchestra.run.smithi028.stdout:checking the maximum length of command line arguments... 1572864 2023-12-16T21:59:34.975 INFO:teuthology.orchestra.run.smithi028.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2023-12-16T21:59:34.975 INFO:teuthology.orchestra.run.smithi028.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2023-12-16T21:59:34.975 INFO:teuthology.orchestra.run.smithi028.stdout:checking for /usr/bin/ld option to reload object files... -r 2023-12-16T21:59:34.975 INFO:teuthology.orchestra.run.smithi028.stdout:checking for objdump... objdump 2023-12-16T21:59:34.976 INFO:teuthology.orchestra.run.smithi028.stdout:checking how to recognize dependent libraries... pass_all 2023-12-16T21:59:34.976 INFO:teuthology.orchestra.run.smithi028.stdout:checking for dlltool... no 2023-12-16T21:59:34.977 INFO:teuthology.orchestra.run.smithi028.stdout:checking how to associate runtime and link libraries... printf %s\n 2023-12-16T21:59:34.977 INFO:teuthology.orchestra.run.smithi028.stdout:checking for ar... ar 2023-12-16T21:59:35.015 INFO:teuthology.orchestra.run.smithi028.stdout:checking for archiver @FILE support... @ 2023-12-16T21:59:35.015 INFO:teuthology.orchestra.run.smithi028.stdout:checking for strip... strip 2023-12-16T21:59:35.016 INFO:teuthology.orchestra.run.smithi028.stdout:checking for ranlib... ranlib 2023-12-16T21:59:35.016 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gawk... gawk 2023-12-16T21:59:35.084 INFO:teuthology.orchestra.run.smithi028.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2023-12-16T21:59:35.086 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sysroot... no 2023-12-16T21:59:35.091 INFO:teuthology.orchestra.run.smithi028.stdout:checking for a working dd... /usr/bin/dd 2023-12-16T21:59:35.095 INFO:teuthology.orchestra.run.smithi028.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2023-12-16T21:59:35.109 INFO:teuthology.orchestra.run.smithi028.stdout:checking for mt... mt 2023-12-16T21:59:35.134 INFO:teuthology.orchestra.run.smithi028.stdout:checking if mt is a manifest tool... no 2023-12-16T21:59:35.167 INFO:teuthology.orchestra.run.smithi028.stdout:checking for stdio.h... yes 2023-12-16T21:59:35.213 INFO:teuthology.orchestra.run.smithi028.stdout:checking for stdlib.h... yes 2023-12-16T21:59:35.261 INFO:teuthology.orchestra.run.smithi028.stdout:checking for string.h... yes 2023-12-16T21:59:35.308 INFO:teuthology.orchestra.run.smithi028.stdout:checking for inttypes.h... yes 2023-12-16T21:59:35.337 INFO:teuthology.orchestra.run.smithi028.stdout:checking for stdint.h... yes 2023-12-16T21:59:35.364 INFO:teuthology.orchestra.run.smithi028.stdout:checking for strings.h... yes 2023-12-16T21:59:35.394 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/stat.h... yes 2023-12-16T21:59:35.422 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/types.h... yes 2023-12-16T21:59:35.460 INFO:teuthology.orchestra.run.smithi028.stdout:checking for unistd.h... yes 2023-12-16T21:59:35.497 INFO:teuthology.orchestra.run.smithi028.stdout:checking for dlfcn.h... yes 2023-12-16T21:59:35.500 INFO:teuthology.orchestra.run.smithi028.stdout:checking for objdir... .libs 2023-12-16T21:59:35.582 INFO:teuthology.orchestra.run.smithi028.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2023-12-16T21:59:35.582 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2023-12-16T21:59:35.608 INFO:teuthology.orchestra.run.smithi028.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2023-12-16T21:59:35.801 INFO:teuthology.orchestra.run.smithi028.stdout:checking if gcc static flag -static works... yes 2023-12-16T21:59:35.830 INFO:teuthology.orchestra.run.smithi028.stdout:checking if gcc supports -c -o file.o... yes 2023-12-16T21:59:35.830 INFO:teuthology.orchestra.run.smithi028.stdout:checking if gcc supports -c -o file.o... (cached) yes 2023-12-16T21:59:35.840 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2023-12-16T21:59:35.866 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether -lc should be explicitly linked in... no 2023-12-16T21:59:35.943 INFO:teuthology.orchestra.run.smithi028.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2023-12-16T21:59:35.943 INFO:teuthology.orchestra.run.smithi028.stdout:checking how to hardcode library paths into programs... immediate 2023-12-16T21:59:35.956 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether stripping libraries is possible... yes 2023-12-16T21:59:35.956 INFO:teuthology.orchestra.run.smithi028.stdout:checking if libtool supports shared libraries... yes 2023-12-16T21:59:35.956 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether to build shared libraries... yes 2023-12-16T21:59:35.956 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether to build static libraries... yes 2023-12-16T21:59:35.966 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gcc... (cached) gcc 2023-12-16T21:59:35.994 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether the compiler supports GNU C... (cached) yes 2023-12-16T21:59:35.994 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2023-12-16T21:59:35.994 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gcc option to enable C11 features... (cached) none needed 2023-12-16T21:59:35.994 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gmake... /usr/bin/gmake 2023-12-16T21:59:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:./configure: line 12754: test: $(SHELL): binary operator expected 2023-12-16T21:59:35.995 INFO:teuthology.orchestra.run.smithi028.stdout:checking for tar... /bin/tar 2023-12-16T21:59:35.995 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gzip... /bin/gzip 2023-12-16T21:59:36.005 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether gcc -MM is supported... yes 2023-12-16T21:59:36.006 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sort... /bin/sort 2023-12-16T21:59:36.006 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether ln -s works... yes 2023-12-16T21:59:36.006 INFO:teuthology.orchestra.run.smithi028.stdout:checking for rpm... no 2023-12-16T21:59:36.039 INFO:teuthology.orchestra.run.smithi028.stdout:checking for assert.h... yes 2023-12-16T21:59:36.081 INFO:teuthology.orchestra.run.smithi028.stdout:checking for bstring.h... no 2023-12-16T21:59:36.115 INFO:teuthology.orchestra.run.smithi028.stdout:checking for libgen.h... yes 2023-12-16T21:59:36.186 INFO:teuthology.orchestra.run.smithi028.stdout:checking for dirent.h... yes 2023-12-16T21:59:36.257 INFO:teuthology.orchestra.run.smithi028.stdout:checking for errno.h... yes 2023-12-16T21:59:36.297 INFO:teuthology.orchestra.run.smithi028.stdout:checking for malloc.h... yes 2023-12-16T21:59:36.335 INFO:teuthology.orchestra.run.smithi028.stdout:checking for uuid.h... no 2023-12-16T21:59:36.385 INFO:teuthology.orchestra.run.smithi028.stdout:checking for uuid/uuid.h... yes 2023-12-16T21:59:36.421 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/uuid.h... no 2023-12-16T21:59:36.455 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/file.h... yes 2023-12-16T21:59:36.489 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/fcntl.h... yes 2023-12-16T21:59:36.525 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/syssgi.h... no 2023-12-16T21:59:36.579 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/param.h... yes 2023-12-16T21:59:36.579 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/stat.h... (cached) yes 2023-12-16T21:59:36.613 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/statvfs.h... yes 2023-12-16T21:59:36.648 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/time.h... yes 2023-12-16T21:59:36.710 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/ioctl.h... yes 2023-12-16T21:59:36.747 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/wait.h... yes 2023-12-16T21:59:36.747 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/types.h... (cached) yes 2023-12-16T21:59:36.747 INFO:teuthology.orchestra.run.smithi028.stdout:checking for strings.h... (cached) yes 2023-12-16T21:59:36.783 INFO:teuthology.orchestra.run.smithi028.stdout:checking for err.h... yes 2023-12-16T21:59:36.817 INFO:teuthology.orchestra.run.smithi028.stdout:checking for linux/falloc.h... yes 2023-12-16T21:59:36.857 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/fs/xfs_fsops.h... no 2023-12-16T21:59:36.893 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/fs/xfs_itable.h... no 2023-12-16T21:59:36.939 INFO:teuthology.orchestra.run.smithi028.stdout:checking for xfs/platform_defs.h... no 2023-12-16T21:59:36.973 INFO:teuthology.orchestra.run.smithi028.stdout:checking for btrfs/ioctl.h... no 2023-12-16T21:59:37.008 INFO:teuthology.orchestra.run.smithi028.stdout:checking for cifs/ioctl.h... no 2023-12-16T21:59:37.043 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/mman.h... yes 2023-12-16T21:59:37.058 INFO:teuthology.orchestra.run.smithi028.stdout:checking for xfs/xfs_log_format.h... no 2023-12-16T21:59:37.101 INFO:teuthology.orchestra.run.smithi028.stdout:checking for uuid_compare... no 2023-12-16T21:59:37.136 INFO:teuthology.orchestra.run.smithi028.stdout:checking for uuid_compare in -luuid... yes 2023-12-16T21:59:37.275 INFO:teuthology.orchestra.run.smithi028.stdout:checking for xfs/xfs.h... yes 2023-12-16T21:59:37.295 INFO:teuthology.orchestra.run.smithi028.stdout:checking for xfs/libxfs.h... no 2023-12-16T21:59:37.338 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gcc options needed to detect all undeclared functions... none needed 2023-12-16T21:59:37.359 INFO:teuthology.orchestra.run.smithi028.stdout:checking whether xlog_assign_lsn is declared... no 2023-12-16T21:59:37.405 INFO:teuthology.orchestra.run.smithi028.stdout:checking for xfs/xqm.h... yes 2023-12-16T21:59:37.470 INFO:teuthology.orchestra.run.smithi028.stdout:checking xfsctl from xfs/xfs.h... ok 2023-12-16T21:59:37.502 INFO:teuthology.orchestra.run.smithi028.stdout:checking for xfs/handle.h... yes 2023-12-16T21:59:37.538 INFO:teuthology.orchestra.run.smithi028.stdout:checking for attr_list_by_handle in -lhandle... yes 2023-12-16T21:59:37.575 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/acl.h... yes 2023-12-16T21:59:37.617 INFO:teuthology.orchestra.run.smithi028.stdout:checking for acl/libacl.h... yes 2023-12-16T21:59:37.653 INFO:teuthology.orchestra.run.smithi028.stdout:checking for acl_init in -lacl... yes 2023-12-16T21:59:37.686 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gdbm-ndbm.h... no 2023-12-16T21:59:37.719 INFO:teuthology.orchestra.run.smithi028.stdout:checking for gdbm/ndbm.h... no 2023-12-16T21:59:37.750 INFO:teuthology.orchestra.run.smithi028.stdout:checking for ndbm.h... no 2023-12-16T21:59:37.783 INFO:teuthology.orchestra.run.smithi028.stdout:checking for libaio.h... yes 2023-12-16T21:59:37.837 INFO:teuthology.orchestra.run.smithi028.stdout:checking for liburing.h... yes 2023-12-16T21:59:37.874 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/capability.h... yes 2023-12-16T21:59:37.907 INFO:teuthology.orchestra.run.smithi028.stdout:checking for linux/fiemap.h... yes 2023-12-16T21:59:37.947 INFO:teuthology.orchestra.run.smithi028.stdout:checking for fallocate... yes 2023-12-16T21:59:37.987 INFO:teuthology.orchestra.run.smithi028.stdout:checking for open_by_handle_at... yes 2023-12-16T21:59:38.019 INFO:teuthology.orchestra.run.smithi028.stdout:checking for sys/prctl.h... yes 2023-12-16T21:59:38.049 INFO:teuthology.orchestra.run.smithi028.stdout:checking for linux/fs.h... yes 2023-12-16T21:59:38.080 INFO:teuthology.orchestra.run.smithi028.stdout:checking for btrfsutil.h... no 2023-12-16T21:59:38.126 INFO:teuthology.orchestra.run.smithi028.stdout:checking for copy_file_range... yes 2023-12-16T21:59:38.168 INFO:teuthology.orchestra.run.smithi028.stdout:checking for SEEK_DATA... yes 2023-12-16T21:59:38.226 INFO:teuthology.orchestra.run.smithi028.stdout:checking for BMV_OF_SHARED... yes 2023-12-16T21:59:38.265 INFO:teuthology.orchestra.run.smithi028.stdout:checking for nftw... yes 2023-12-16T21:59:38.309 INFO:teuthology.orchestra.run.smithi028.stdout:checking for RLIMIT_NOFILE... yes 2023-12-16T21:59:38.350 INFO:teuthology.orchestra.run.smithi028.stdout:checking for XFS_IOC_EXCHANGE_RANGE... no 2023-12-16T21:59:38.385 INFO:teuthology.orchestra.run.smithi028.stdout:checking for FICLONE... yes 2023-12-16T21:59:38.421 INFO:teuthology.orchestra.run.smithi028.stdout:checking for renameat2... yes 2023-12-16T21:59:38.458 INFO:teuthology.orchestra.run.smithi028.stdout:checking for reallocarray... yes 2023-12-16T21:59:38.489 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct mount_attr... yes 2023-12-16T21:59:38.536 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct btrfs_qgroup_limit... yes 2023-12-16T21:59:38.569 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct btrfs_qgroup_inherit... yes 2023-12-16T21:59:38.602 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct btrfs_ioctl_vol_args... yes 2023-12-16T21:59:38.635 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct btrfs_ioctl_vol_args_v2... yes 2023-12-16T21:59:38.669 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct btrfs_ioctl_ino_lookup_args... yes 2023-12-16T21:59:38.703 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct btrfs_ioctl_ino_lookup_user_args... yes 2023-12-16T21:59:38.744 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct btrfs_ioctl_get_subvol_rootref_args... yes 2023-12-16T21:59:38.784 INFO:teuthology.orchestra.run.smithi028.stdout:checking for linux/btrfs.h... yes 2023-12-16T21:59:38.820 INFO:teuthology.orchestra.run.smithi028.stdout:checking for linux/btrfs_tree.h... yes 2023-12-16T21:59:38.840 INFO:teuthology.orchestra.run.smithi028.stdout:checking for struct btrfs_ioctl_vol_args_v2.subvolid... yes 2023-12-16T21:59:38.858 INFO:teuthology.orchestra.run.smithi028.stdout:configure: creating ./config.status 2023-12-16T21:59:39.296 INFO:teuthology.orchestra.run.smithi028.stdout:config.status: creating include/builddefs 2023-12-16T21:59:39.312 INFO:teuthology.orchestra.run.smithi028.stdout:config.status: creating include/config.h 2023-12-16T21:59:39.324 INFO:teuthology.orchestra.run.smithi028.stdout:config.status: executing libtool commands 2023-12-16T21:59:39.371 INFO:teuthology.orchestra.run.smithi028.stdout:Building include 2023-12-16T21:59:39.375 INFO:teuthology.orchestra.run.smithi028.stdout:Building lib 2023-12-16T21:59:39.379 INFO:teuthology.orchestra.run.smithi028.stdout: [LTDEP] 2023-12-16T21:59:39.530 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] dataascii.lo 2023-12-16T21:59:39.641 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] databin.lo 2023-12-16T21:59:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] datapid.lo 2023-12-16T21:59:39.915 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] file_lock.lo 2023-12-16T21:59:40.075 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] forker.lo 2023-12-16T21:59:40.228 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] pattern.lo 2023-12-16T21:59:40.353 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] open_flags.lo 2023-12-16T21:59:40.506 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] random_range.lo 2023-12-16T21:59:40.760 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] string_to_tokens.lo 2023-12-16T21:59:40.867 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] str_to_bytes.lo 2023-12-16T21:59:41.010 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] tlibio.lo 2023-12-16T21:59:41.279 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] write_log.lo 2023-12-16T21:59:41.352 INFO:teuthology.orchestra.run.smithi028.stderr:write_log.c: In function 'wlog_record_write': 2023-12-16T21:59:41.352 INFO:teuthology.orchestra.run.smithi028.stderr:write_log.c:205:13: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:41.352 INFO:teuthology.orchestra.run.smithi028.stderr: 205 | write(wfile->w_afd, wbuf, reclen); 2023-12-16T21:59:41.352 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:41.352 INFO:teuthology.orchestra.run.smithi028.stderr:write_log.c:209:13: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:41.352 INFO:teuthology.orchestra.run.smithi028.stderr: 209 | write(wfile->w_rfd, wbuf, reclen); 2023-12-16T21:59:41.352 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] random.lo 2023-12-16T21:59:41.671 INFO:teuthology.orchestra.run.smithi028.stdout: [LD] 2023-12-16T21:59:41.747 INFO:teuthology.orchestra.run.smithi028.stdout:Building ltp 2023-12-16T21:59:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: [DEP] 2023-12-16T21:59:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] doio 2023-12-16T21:59:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fsstress 2023-12-16T21:59:44.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:59:44.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:59:44.141 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fsx 2023-12-16T21:59:44.951 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] growfiles 2023-12-16T21:59:45.531 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] iogen 2023-12-16T21:59:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:iogen.c: In function 'main': 2023-12-16T21:59:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:iogen.c:280:9: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:45.661 INFO:teuthology.orchestra.run.smithi028.stderr: 280 | write(outfd, (char *)&req, sizeof(req)); 2023-12-16T21:59:45.661 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:45.664 INFO:teuthology.orchestra.run.smithi028.stderr:iogen.c: In function 'parse_cmdline': 2023-12-16T21:59:45.664 INFO:teuthology.orchestra.run.smithi028.stderr:iogen.c:1535:17: warning: ignoring return value of 'getcwd' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:45.664 INFO:teuthology.orchestra.run.smithi028.stderr: 1535 | getcwd(fptr->f_path, 2023-12-16T21:59:45.664 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:45.664 INFO:teuthology.orchestra.run.smithi028.stderr: 1536 | sizeof(fptr->f_path)-1); 2023-12-16T21:59:45.664 INFO:teuthology.orchestra.run.smithi028.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:45.961 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-stress 2023-12-16T21:59:46.091 INFO:teuthology.orchestra.run.smithi028.stderr:aio-stress.c: In function 'setup_shared_mem': 2023-12-16T21:59:46.091 INFO:teuthology.orchestra.run.smithi028.stderr:aio-stress.c:1013:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:46.091 INFO:teuthology.orchestra.run.smithi028.stderr: 1013 | ftruncate(fd, total_ram); 2023-12-16T21:59:46.091 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:46.432 INFO:teuthology.orchestra.run.smithi028.stdout:Building src 2023-12-16T21:59:46.440 INFO:teuthology.orchestra.run.smithi028.stdout: [DEP] 2023-12-16T21:59:48.594 INFO:teuthology.orchestra.run.smithi028.stdout: [DEP] 2023-12-16T21:59:48.633 INFO:teuthology.orchestra.run.smithi028.stdout: [DEP] 2023-12-16T21:59:48.641 INFO:teuthology.orchestra.run.smithi028.stdout: [DEP] 2023-12-16T21:59:48.927 INFO:teuthology.orchestra.run.smithi028.stdout: [DEP] 2023-12-16T21:59:48.956 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] dirstress 2023-12-16T21:59:49.215 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fill 2023-12-16T21:59:49.374 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fill2 2023-12-16T21:59:49.605 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] getpagesize 2023-12-16T21:59:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] holes 2023-12-16T21:59:49.942 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] lstat64 2023-12-16T21:59:50.132 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] nametest 2023-12-16T21:59:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:nametest.c: In function 'main': 2023-12-16T21:59:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:nametest.c:236:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:50.243 INFO:teuthology.orchestra.run.smithi028.stderr: 236 | write(1, "\n", 1); 2023-12-16T21:59:50.243 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T21:59:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:nametest.c:238:25: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:50.243 INFO:teuthology.orchestra.run.smithi028.stderr: 238 | write(1, ".", 1); 2023-12-16T21:59:50.244 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T21:59:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:nametest.c:288:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:50.244 INFO:teuthology.orchestra.run.smithi028.stderr: 288 | write(1, ".", 1); 2023-12-16T21:59:50.244 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T21:59:50.372 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] permname 2023-12-16T21:59:50.572 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] randholes 2023-12-16T21:59:50.829 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] runas 2023-12-16T21:59:51.024 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] truncfile 2023-12-16T21:59:51.203 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] usemem 2023-12-16T21:59:51.347 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] mmapcat 2023-12-16T21:59:51.443 INFO:teuthology.orchestra.run.smithi028.stderr:mmapcat.c: In function 'main': 2023-12-16T21:59:51.443 INFO:teuthology.orchestra.run.smithi028.stderr:mmapcat.c:29:9: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:51.444 INFO:teuthology.orchestra.run.smithi028.stderr: 29 | write(1,ptr2,ptr - ptr2); 2023-12-16T21:59:51.444 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:51.476 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] append_reader 2023-12-16T21:59:51.610 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] append_writer 2023-12-16T21:59:51.747 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] dirperf 2023-12-16T21:59:51.970 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] metaperf 2023-12-16T21:59:52.101 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c: In function 'crfiles': 2023-12-16T21:59:52.102 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c:166:25: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:52.102 INFO:teuthology.orchestra.run.smithi028.stderr: 166 | write(fd, buf, fsize); 2023-12-16T21:59:52.102 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:52.103 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c: In function 't_linkun': 2023-12-16T21:59:52.103 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c:454:25: warning: ignoring return value of 'link' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:52.103 INFO:teuthology.orchestra.run.smithi028.stderr: 454 | link("a", *fnp); 2023-12-16T21:59:52.103 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~ 2023-12-16T21:59:52.103 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c: In function 't_chown': 2023-12-16T21:59:52.103 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c:404:33: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:52.103 INFO:teuthology.orchestra.run.smithi028.stderr: 404 | chown(*fnp, 2, -1); 2023-12-16T21:59:52.103 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c:406:33: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr: 406 | chown(*fnp, 1, -1); 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c: In function 'i_chown': 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c:291:17: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr: 291 | chown(*fnp, 1, -1); 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c: In function 'main': 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:metaperf.c:152:9: warning: ignoring return value of 'chdir' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr: 152 | chdir(".."); 2023-12-16T21:59:52.104 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~ 2023-12-16T21:59:52.283 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] devzero 2023-12-16T21:59:52.468 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] feature 2023-12-16T21:59:52.699 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] alloc 2023-12-16T21:59:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:alloc.c: In function 'main': 2023-12-16T21:59:52.825 INFO:teuthology.orchestra.run.smithi028.stderr:alloc.c:247:17: warning: ignoring return value of 'mkstemp' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:52.825 INFO:teuthology.orchestra.run.smithi028.stderr: 247 | mkstemp(tmpfile); 2023-12-16T21:59:52.825 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T21:59:52.923 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fault 2023-12-16T21:59:53.088 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fstest 2023-12-16T21:59:53.349 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_access_root 2023-12-16T21:59:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:t_access_root.c: In function 'main': 2023-12-16T21:59:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:t_access_root.c:55:5: warning: ignoring return value of 'system' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:53.467 INFO:teuthology.orchestra.run.smithi028.stderr: 55 | system(cmd); 2023-12-16T21:59:53.467 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~ 2023-12-16T21:59:53.515 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] godown 2023-12-16T21:59:53.694 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] resvtest 2023-12-16T21:59:53.884 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] writemod 2023-12-16T21:59:54.032 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] writev_on_pagefault 2023-12-16T21:59:54.195 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] makeextents 2023-12-16T21:59:54.385 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] itrash 2023-12-16T21:59:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] rename 2023-12-16T21:59:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] multi_open_unlink 2023-12-16T21:59:54.823 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] unwritten_sync 2023-12-16T21:59:55.017 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] genhashnames 2023-12-16T21:59:55.222 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_holes 2023-12-16T21:59:55.374 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_writev 2023-12-16T21:59:55.515 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_truncate_cmtime 2023-12-16T21:59:55.660 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] dirhash_collide 2023-12-16T21:59:55.840 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_rename_overwrite 2023-12-16T21:59:55.976 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] holetest 2023-12-16T21:59:56.088 INFO:teuthology.orchestra.run.smithi028.stderr:holetest.c: In function 'pt_write_marker': 2023-12-16T21:59:56.088 INFO:teuthology.orchestra.run.smithi028.stderr:holetest.c:140:17: warning: ignoring return value of 'pwrite' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:56.088 INFO:teuthology.orchestra.run.smithi028.stderr: 140 | pwrite(fd, &tid, sizeof(tid), i * page_size + pgoff); 2023-12-16T21:59:56.088 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:56.208 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_truncate_self 2023-12-16T21:59:56.345 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] af_unix 2023-12-16T21:59:56.484 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_stale_pmd 2023-12-16T21:59:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:t_mmap_stale_pmd.c: In function 'main': 2023-12-16T21:59:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:t_mmap_stale_pmd.c:42:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:56.591 INFO:teuthology.orchestra.run.smithi028.stderr: 42 | ftruncate(fd, 0); 2023-12-16T21:59:56.591 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T21:59:56.592 INFO:teuthology.orchestra.run.smithi028.stderr:t_mmap_stale_pmd.c:43:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:56.592 INFO:teuthology.orchestra.run.smithi028.stderr: 43 | ftruncate(fd, MiB(4)); 2023-12-16T21:59:56.592 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:56.592 INFO:teuthology.orchestra.run.smithi028.stderr:t_mmap_stale_pmd.c:55:9: warning: ignoring return value of 'pwrite' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:56.592 INFO:teuthology.orchestra.run.smithi028.stderr: 55 | pwrite(fd, buffer, strlen(buffer), MiB(2)); 2023-12-16T21:59:56.592 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:56.628 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_cow_race 2023-12-16T21:59:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:t_mmap_cow_race.c: In function 'main': 2023-12-16T21:59:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:t_mmap_cow_race.c:61:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:56.732 INFO:teuthology.orchestra.run.smithi028.stderr: 61 | ftruncate(fd, 0); 2023-12-16T21:59:56.732 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T21:59:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:t_mmap_cow_race.c:62:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:56.733 INFO:teuthology.orchestra.run.smithi028.stderr: 62 | ftruncate(fd, MiB(2)); 2023-12-16T21:59:56.733 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2023-12-16T21:59:56.784 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_fallocate 2023-12-16T21:59:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fsync-err 2023-12-16T21:59:57.152 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_write_ro 2023-12-16T21:59:57.311 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_ext4_dax_journal_corruption 2023-12-16T21:59:57.431 INFO:teuthology.orchestra.run.smithi028.stderr:t_ext4_dax_journal_corruption.c: In function 'main': 2023-12-16T21:59:57.432 INFO:teuthology.orchestra.run.smithi028.stderr:t_ext4_dax_journal_corruption.c:61:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T21:59:57.432 INFO:teuthology.orchestra.run.smithi028.stderr: 61 | ftruncate(fd, 0); 2023-12-16T21:59:57.432 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T21:59:57.478 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_ext4_dax_inline_corruption 2023-12-16T21:59:57.644 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_ofd_locks 2023-12-16T21:59:57.846 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_collision 2023-12-16T21:59:58.068 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] mmap-write-concurrent 2023-12-16T21:59:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_get_file_time 2023-12-16T21:59:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_create_short_dirs 2023-12-16T21:59:58.549 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_create_long_dirs 2023-12-16T21:59:58.713 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_enospc 2023-12-16T21:59:58.908 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_writev_overlap 2023-12-16T21:59:59.076 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] checkpoint_journal 2023-12-16T21:59:59.244 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] mmap-rw-fault 2023-12-16T21:59:59.442 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] allocstale 2023-12-16T21:59:59.625 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_cow_memory_failure 2023-12-16T21:59:59.797 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fake-dump-rootino 2023-12-16T21:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] dio-buf-fault 2023-12-16T22:00:00.112 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] rewinddir-test 2023-12-16T22:00:00.282 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] xfsctl 2023-12-16T22:00:00.478 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] bstat 2023-12-16T22:00:00.712 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mtab 2023-12-16T22:00:00.836 INFO:teuthology.orchestra.run.smithi028.stderr:t_mtab.c: In function 'update_mtab': 2023-12-16T22:00:00.836 INFO:teuthology.orchestra.run.smithi028.stderr:t_mtab.c:256:17: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:00.836 INFO:teuthology.orchestra.run.smithi028.stderr: 256 | chown (mounted_temp, sbuf.st_uid, sbuf.st_gid); 2023-12-16T22:00:00.836 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] getdevicesize 2023-12-16T22:00:01.080 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] preallo_rw_pattern_reader 2023-12-16T22:00:01.197 INFO:teuthology.orchestra.run.smithi028.stderr:In file included from preallo_rw_pattern_reader.c:2: 2023-12-16T22:00:01.197 INFO:teuthology.orchestra.run.smithi028.stderr:iopat.c: In function 'main': 2023-12-16T22:00:01.198 INFO:teuthology.orchestra.run.smithi028.stderr:iopat.c:38:4: warning: ignoring return value of 'read' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:01.198 INFO:teuthology.orchestra.run.smithi028.stderr: 38 | read(fd, &x, 1048576); 2023-12-16T22:00:01.198 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:01.232 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] preallo_rw_pattern_writer 2023-12-16T22:00:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:In file included from preallo_rw_pattern_writer.c:2: 2023-12-16T22:00:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:iopat.c: In function 'main': 2023-12-16T22:00:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:iopat.c:34:4: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:01.350 INFO:teuthology.orchestra.run.smithi028.stderr: 34 | write(fd, &x, 1048576); 2023-12-16T22:00:01.350 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:01.379 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] ftrunc 2023-12-16T22:00:01.515 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] trunc 2023-12-16T22:00:01.671 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fs_perms 2023-12-16T22:00:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:fs_perms.c: In function 'main': 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:fs_perms.c:54:4: warning: ignoring return value of 'system' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr: 54 | system("rm -f test.file"); 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:fs_perms.c: In function 'testsetup': 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:fs_perms.c:67:4: warning: ignoring return value of 'system' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr: 67 | system("cp testx.file test.file"); 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:fs_perms.c:69:4: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr: 69 | chown("test.file",cuserId,cgroupId); 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:fs_perms.c: In function 'testfperm': 2023-12-16T22:00:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:fs_perms.c:123:5: warning: ignoring return value of 'seteuid' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:01.777 INFO:teuthology.orchestra.run.smithi028.stderr: 123 | seteuid(0); 2023-12-16T22:00:01.777 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~ 2023-12-16T22:00:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:fs_perms.c:124:5: warning: ignoring return value of 'setegid' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:01.777 INFO:teuthology.orchestra.run.smithi028.stderr: 124 | setegid(0); 2023-12-16T22:00:01.777 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~ 2023-12-16T22:00:01.819 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] testx 2023-12-16T22:00:01.935 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] looptest 2023-12-16T22:00:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] locktest 2023-12-16T22:00:02.460 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] unwritten_mmap 2023-12-16T22:00:02.630 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] bulkstat_unlink_test 2023-12-16T22:00:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:bulkstat_unlink_test.c: In function 'main': 2023-12-16T22:00:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:bulkstat_unlink_test.c:106:25: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:02.749 INFO:teuthology.orchestra.run.smithi028.stderr: 106 | write(fd[i], fname, sizeof(fname)); 2023-12-16T22:00:02.749 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:bulkstat_unlink_test.c:178:17: warning: ignoring return value of 'system' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:02.749 INFO:teuthology.orchestra.run.smithi028.stderr: 178 | system(fname); 2023-12-16T22:00:02.749 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~ 2023-12-16T22:00:02.838 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] deduperace 2023-12-16T22:00:03.049 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] bulkstat_unlink_test_modified 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:bulkstat_unlink_test_modified.c: In function 'main': 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:bulkstat_unlink_test_modified.c:83:13: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr: 83 | write(fd, fname, sizeof(fname)); 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:bulkstat_unlink_test_modified.c:139:13: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr: 139 | write(fd, fname, sizeof(fname)); 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:bulkstat_unlink_test_modified.c:185:9: warning: ignoring return value of 'system' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.177 INFO:teuthology.orchestra.run.smithi028.stderr: 185 | system(fname); 2023-12-16T22:00:03.178 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~ 2023-12-16T22:00:03.342 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_dir_offset 2023-12-16T22:00:03.501 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_futimens 2023-12-16T22:00:03.648 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_immutable 2023-12-16T22:00:03.789 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c: In function 'create_test_area': 2023-12-16T22:00:03.789 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:2061:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.789 INFO:teuthology.orchestra.run.smithi028.stderr: 2061 | asprintf(&path, "%s/immutable.d", dir); 2023-12-16T22:00:03.789 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.789 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:2096:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.789 INFO:teuthology.orchestra.run.smithi028.stderr: 2096 | asprintf(&path, "%s/append-only.d", dir); 2023-12-16T22:00:03.789 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c: In function 'remove_test_area': 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:2152:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr: 2152 | asprintf(&path, "%s/immutable.d", dir); 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:2163:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr: 2163 | asprintf(&path, "%s/empty-immutable.d", dir); 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:2175:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.790 INFO:teuthology.orchestra.run.smithi028.stderr: 2175 | asprintf(&path, "%s/append-only.d", dir); 2023-12-16T22:00:03.791 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.791 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:2186:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.791 INFO:teuthology.orchestra.run.smithi028.stderr: 2186 | asprintf(&path, "%s/empty-append-only.d", dir); 2023-12-16T22:00:03.791 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.792 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:2197:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.792 INFO:teuthology.orchestra.run.smithi028.stderr: 2197 | asprintf(&path, "%s/append-only.f", dir); 2023-12-16T22:00:03.792 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.792 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:2209:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.792 INFO:teuthology.orchestra.run.smithi028.stderr: 2209 | asprintf(&path, "%s/immutable.f", dir); 2023-12-16T22:00:03.792 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c: In function 'check_test_area': 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1788:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr: 1788 | asprintf(&path, "%s/", dir); 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1800:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr: 1800 | asprintf(&path, "%s/immutable.f", dir); 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1812:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr: 1812 | asprintf(&path, "%s/append-only.f", dir); 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1824:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr: 1824 | asprintf(&path, "%s/immutable.d", dir); 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.793 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1837:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: 1837 | asprintf(&path, "%s/append-only.d", dir); 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1850:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: 1850 | asprintf(&path, "%s/immutable.d/file", dir); 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1862:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: 1862 | asprintf(&path, "%s/append-only.d/file", dir); 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1874:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: 1874 | asprintf(&path, "%s/immutable.d/dir", dir); 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1887:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: 1887 | asprintf(&path, "%s/append-only.d/dir", dir); 2023-12-16T22:00:03.794 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.796 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c: In function 'test_immutable': 2023-12-16T22:00:03.796 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:151:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.796 INFO:teuthology.orchestra.run.smithi028.stderr: 151 | asprintf(&path, "%s/immutable.f", dir); 2023-12-16T22:00:03.796 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.796 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:381:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.796 INFO:teuthology.orchestra.run.smithi028.stderr: 381 | asprintf(&linkpath, "%s/immutable.f.hardlink", dir); 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:408:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: 408 | chown(path, 0, 0); 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:485:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: 485 | asprintf(&linkpath, "%s/immutable.f.newname", dir); 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:532:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: 532 | asprintf(&path, "%s/immutable.d/file", dir); 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:553:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: 553 | chown(path, 0, 0); 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:556:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.797 INFO:teuthology.orchestra.run.smithi028.stderr: 556 | asprintf(&linkpath, "%s/immutable.d/file.link", dir); 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:575:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: 575 | asprintf(&linkpath, "%s/immutable.d/file.newname", dir); 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:595:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: 595 | asprintf(&path, "%s/immutable.d/newfile", dir); 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:619:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: 619 | asprintf(&path, "%s/immutable.d/newdir", dir); 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:631:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: 631 | asprintf(&path, "%s/immutable.d/dir/newfile-%d", dir, getuid()); 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:653:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: 653 | chown(path, 0, 0); 2023-12-16T22:00:03.798 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:657:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr: 657 | asprintf(&path, "%s/immutable.d/dir", dir); 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:668:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr: 668 | asprintf(&path, "%s/immutable.d", dir); 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:705:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr: 705 | chown(path, 0, 0); 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:782:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr: 782 | asprintf(&path, "%s/empty-immutable.d", dir); 2023-12-16T22:00:03.799 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.801 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c: In function 'test_append': 2023-12-16T22:00:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:822:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.802 INFO:teuthology.orchestra.run.smithi028.stderr: 822 | asprintf(&path, "%s/append-only.f", dir); 2023-12-16T22:00:03.802 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1024:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.802 INFO:teuthology.orchestra.run.smithi028.stderr: 1024 | asprintf(&linkpath, "%s/append-only.f.hardlink", dir); 2023-12-16T22:00:03.802 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1051:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr: 1051 | chown(path, 0, 0); 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1126:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr: 1126 | asprintf(&linkpath, "%s/append-only.f.newname", dir); 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1495:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr: 1495 | asprintf(&path, "%s/append-only.d/file", dir); 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1516:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.804 INFO:teuthology.orchestra.run.smithi028.stderr: 1516 | chown(path, 0, 0); 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1519:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: 1519 | asprintf(&linkpath, "%s/append-only.d/file.link-%d", dir, getuid()); 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1529:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: 1529 | asprintf(&linkpath, "%s/append-only.d/file.symlink-%d", dir, getuid()); 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1539:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: 1539 | asprintf(&linkpath, "%s/append-only.d/file.newname", dir); 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1559:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: 1559 | asprintf(&path, "%s/append-only.d/newfile-%d", dir, getuid()); 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1573:11: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: 1573 | asprintf(&path, "%s/append-only.d/newdev-%d", dir, getuid()); 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1586:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: 1586 | asprintf(&path, "%s/append-only.d/newdir-%d", dir, getuid()); 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1596:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: 1596 | asprintf(&path, "%s/append-only.d/newdir-%d/newfile-%d", dir, getuid(), getuid()); 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1618:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: 1618 | chown(path, 0, 0); 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1622:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: 1622 | asprintf(&path, "%s/append-only.d/dir/newfile-%d", dir, getuid()); 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.806 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1644:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: 1644 | chown(path, 0, 0); 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1648:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: 1648 | asprintf(&path, "%s/append-only.d/dir", dir); 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1659:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: 1659 | asprintf(&path, "%s/append-only.d", dir); 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1692:16: warning: ignoring return value of 'chown' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: 1692 | chown(path, 0, 0); 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:t_immutable.c:1769:6: warning: ignoring return value of 'asprintf' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: 1769 | asprintf(&path, "%s/empty-append-only.d", dir); 2023-12-16T22:00:03.807 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:04.382 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] stale_handle 2023-12-16T22:00:04.495 INFO:teuthology.orchestra.run.smithi028.stderr:stale_handle.c: In function 'main': 2023-12-16T22:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr:stale_handle.c:102:9: warning: ignoring return value of 'system' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr: 102 | system("echo 3 > /proc/sys/vm/drop_caches"); 2023-12-16T22:00:04.496 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:04.548 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] pwrite_mmap_blocked 2023-12-16T22:00:04.692 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_dir_offset2 2023-12-16T22:00:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] seek_sanity_test 2023-12-16T22:00:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c: In function 'test18': 2023-12-16T22:00:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c:460:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.037 INFO:teuthology.orchestra.run.smithi028.stderr: 460 | ftruncate(fd, 0); 2023-12-16T22:00:05.037 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c: In function 'test_basic_support': 2023-12-16T22:00:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c:1207:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.040 INFO:teuthology.orchestra.run.smithi028.stderr: 1207 | ftruncate(fd, 0); 2023-12-16T22:00:05.040 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c:1256:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.040 INFO:teuthology.orchestra.run.smithi028.stderr: 1256 | ftruncate(fd, 0); 2023-12-16T22:00:05.040 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c: In function 'get_io_sizes': 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c:88:17: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr: 88 | ftruncate(fd, 0); 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c:89:17: warning: ignoring return value of 'pwrite' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr: 89 | pwrite(fd, "a", 1, offset); 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c:98:17: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr: 98 | ftruncate(fd, 0); 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:seek_sanity_test.c:99:17: warning: ignoring return value of 'pwrite' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.041 INFO:teuthology.orchestra.run.smithi028.stderr: 99 | pwrite(fd, "a", 1, offset); 2023-12-16T22:00:05.042 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.382 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] seek_copy_test 2023-12-16T22:00:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_readdir_1 2023-12-16T22:00:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:t_readdir_1.c: In function 'main': 2023-12-16T22:00:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:t_readdir_1.c:30:25: warning: ignoring return value of 'read' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.677 INFO:teuthology.orchestra.run.smithi028.stderr: 30 | read(fd, buf, 100); 2023-12-16T22:00:05.677 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_readdir_2 2023-12-16T22:00:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fsync-tester 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr:fsync-tester.c: In function 'main': 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr:fsync-tester.c:391:17: warning: ignoring return value of 'read' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr: 391 | read(fd, &seed, sizeof(seed)); 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr:fsync-tester.c:541:17: warning: ignoring return value of 'system' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr: 541 | system("reboot -fn"); 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr:fsync-tester.c: In function 'drop_all_caches': 2023-12-16T22:00:05.950 INFO:teuthology.orchestra.run.smithi028.stderr:fsync-tester.c:90:9: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:05.951 INFO:teuthology.orchestra.run.smithi028.stderr: 90 | write(fd, value, sizeof(value)-1); 2023-12-16T22:00:05.951 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] nsexec 2023-12-16T22:00:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] cloner 2023-12-16T22:00:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] renameat2 2023-12-16T22:00:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_getcwd 2023-12-16T22:00:06.767 INFO:teuthology.orchestra.run.smithi028.stderr:t_getcwd.c: In function 'test_getcwd': 2023-12-16T22:00:06.767 INFO:teuthology.orchestra.run.smithi028.stderr:t_getcwd.c:21:17: warning: ignoring return value of 'getcwd' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:06.767 INFO:teuthology.orchestra.run.smithi028.stderr: 21 | getcwd(cur_cwd, BUF_SIZE); 2023-12-16T22:00:06.767 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] e4compact 2023-12-16T22:00:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] test-nextquota 2023-12-16T22:00:07.193 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] punch-alternating 2023-12-16T22:00:07.360 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] attr-list-by-handle-cursor-test 2023-12-16T22:00:07.533 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] listxattr 2023-12-16T22:00:07.681 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] dio-interleaved 2023-12-16T22:00:07.836 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_dir_type 2023-12-16T22:00:07.984 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] dio-invalidate-cache 2023-12-16T22:00:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:dio-invalidate-cache.c: In function 'main': 2023-12-16T22:00:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:dio-invalidate-cache.c:286:17: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:08.092 INFO:teuthology.orchestra.run.smithi028.stderr: 286 | write(fd, buf, blksz * nr_child); 2023-12-16T22:00:08.092 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] stat_test 2023-12-16T22:00:08.505 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_encrypted_d_revalidate 2023-12-16T22:00:08.672 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] attr_replace_test 2023-12-16T22:00:08.848 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] swapon 2023-12-16T22:00:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] mkswap 2023-12-16T22:00:09.182 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_attr_corruption 2023-12-16T22:00:09.340 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_open_tmpfiles 2023-12-16T22:00:09.528 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fscrypt-crypt-util 2023-12-16T22:00:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] bulkstat_null_ocount 2023-12-16T22:00:10.379 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] splice-test 2023-12-16T22:00:10.569 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] chprojid_fail 2023-12-16T22:00:10.720 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] detached_mounts_propagation 2023-12-16T22:00:10.911 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] ext4_resize 2023-12-16T22:00:11.074 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_readdir_3 2023-12-16T22:00:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] splice2pipe 2023-12-16T22:00:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:splice2pipe.c: In function 'prepare_pipe': 2023-12-16T22:00:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:splice2pipe.c:57:17: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:11.372 INFO:teuthology.orchestra.run.smithi028.stderr: 57 | write(p[1], buffer, n); 2023-12-16T22:00:11.372 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:splice2pipe.c:65:17: warning: ignoring return value of 'read' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:11.373 INFO:teuthology.orchestra.run.smithi028.stderr: 65 | read(p[0], buffer, n); 2023-12-16T22:00:11.373 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:11.424 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] uuid_ioctl 2023-12-16T22:00:11.582 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fiemap-tester 2023-12-16T22:00:11.856 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_stripealign 2023-12-16T22:00:12.013 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] open_by_handle 2023-12-16T22:00:12.284 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_mmap_dio 2023-12-16T22:00:12.464 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] uring_read_fault 2023-12-16T22:00:12.657 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] xfsfind 2023-12-16T22:00:12.866 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] t_reflink_read_race 2023-12-16T22:00:13.074 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] fssum 2023-12-16T22:00:13.554 INFO:teuthology.orchestra.run.smithi028.stdout:Building log-writes 2023-12-16T22:00:13.560 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] replay-log 2023-12-16T22:00:13.908 INFO:teuthology.orchestra.run.smithi028.stdout:Building perf 2023-12-16T22:00:13.913 INFO:teuthology.orchestra.run.smithi028.stdout:Building aio-dio-regress 2023-12-16T22:00:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aiocp 2023-12-16T22:00:14.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:00:14.147 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:00:14.163 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-append-write-fallocate-race 2023-12-16T22:00:14.342 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-append-write-read-race 2023-12-16T22:00:14.544 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-cow-race 2023-12-16T22:00:14.704 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-cycle-write 2023-12-16T22:00:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:aio-dio-cycle-write.c: In function 'init_test': 2023-12-16T22:00:14.820 INFO:teuthology.orchestra.run.smithi028.stderr:aio-dio-cycle-write.c:83:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:14.820 INFO:teuthology.orchestra.run.smithi028.stderr: 83 | ftruncate(fd, buf_size); 2023-12-16T22:00:14.820 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-eof-race 2023-12-16T22:00:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-extend-stat 2023-12-16T22:00:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-fcntl-race 2023-12-16T22:00:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-hole-filling-race 2023-12-16T22:00:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-invalidate-failure 2023-12-16T22:00:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-invalidate-readahead 2023-12-16T22:00:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aiodio_sparse2 2023-12-16T22:00:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:aiodio_sparse2.c: In function 'aiodio_sparse': 2023-12-16T22:00:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:aiodio_sparse2.c:111:17: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:16.101 INFO:teuthology.orchestra.run.smithi028.stderr: 111 | ftruncate(fd, filesize); 2023-12-16T22:00:16.101 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:aiodio_sparse2.c: In function 'dirty_freeblocks': 2023-12-16T22:00:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:aiodio_sparse2.c:274:9: warning: ignoring return value of 'ftruncate' declared with attribute 'warn_unused_result' [-Wunused-result] 2023-12-16T22:00:16.102 INFO:teuthology.orchestra.run.smithi028.stderr: 274 | ftruncate(fd, size); 2023-12-16T22:00:16.102 INFO:teuthology.orchestra.run.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-12-16T22:00:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-subblock-eof-read 2023-12-16T22:00:16.395 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-dio-write-verify 2023-12-16T22:00:16.615 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-free-ring-with-bogus-nr-pages 2023-12-16T22:00:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-io-setup-with-nonwritable-context-pointer 2023-12-16T22:00:16.892 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] aio-last-ref-held-by-io 2023-12-16T22:00:17.072 INFO:teuthology.orchestra.run.smithi028.stdout:Building vfs 2023-12-16T22:00:17.079 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] vfstest 2023-12-16T22:00:21.469 INFO:teuthology.orchestra.run.smithi028.stdout: [CC] mount-idmapped 2023-12-16T22:00:21.945 INFO:teuthology.orchestra.run.smithi028.stdout:Building m4 2023-12-16T22:00:21.950 INFO:teuthology.orchestra.run.smithi028.stdout:Building common 2023-12-16T22:00:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:Building tests 2023-12-16T22:00:21.964 INFO:teuthology.orchestra.run.smithi028.stdout:Building btrfs 2023-12-16T22:00:21.971 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/btrfs/group.list 2023-12-16T22:00:22.043 INFO:teuthology.orchestra.run.smithi028.stdout:Building ceph 2023-12-16T22:00:22.048 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ceph/group.list 2023-12-16T22:00:22.067 INFO:teuthology.orchestra.run.smithi028.stdout:Building cifs 2023-12-16T22:00:22.073 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/cifs/group.list 2023-12-16T22:00:22.087 INFO:teuthology.orchestra.run.smithi028.stdout:Building ext4 2023-12-16T22:00:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ext4/group.list 2023-12-16T22:00:22.136 INFO:teuthology.orchestra.run.smithi028.stdout:Building f2fs 2023-12-16T22:00:22.142 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/f2fs/group.list 2023-12-16T22:00:22.163 INFO:teuthology.orchestra.run.smithi028.stdout:Building generic 2023-12-16T22:00:22.170 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/generic/group.list 2023-12-16T22:00:22.285 INFO:teuthology.orchestra.run.smithi028.stdout:Building nfs 2023-12-16T22:00:22.292 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/nfs/group.list 2023-12-16T22:00:22.310 INFO:teuthology.orchestra.run.smithi028.stdout:Building ocfs2 2023-12-16T22:00:22.317 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ocfs2/group.list 2023-12-16T22:00:22.334 INFO:teuthology.orchestra.run.smithi028.stdout:Building overlay 2023-12-16T22:00:22.339 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/overlay/group.list 2023-12-16T22:00:22.387 INFO:teuthology.orchestra.run.smithi028.stdout:Building perf 2023-12-16T22:00:22.393 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/perf/group.list 2023-12-16T22:00:22.405 INFO:teuthology.orchestra.run.smithi028.stdout:Building selftest 2023-12-16T22:00:22.411 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/selftest/group.list 2023-12-16T22:00:22.426 INFO:teuthology.orchestra.run.smithi028.stdout:Building shared 2023-12-16T22:00:22.431 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/shared/group.list 2023-12-16T22:00:22.451 INFO:teuthology.orchestra.run.smithi028.stdout:Building tmpfs 2023-12-16T22:00:22.457 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/tmpfs/group.list 2023-12-16T22:00:22.472 INFO:teuthology.orchestra.run.smithi028.stdout:Building udf 2023-12-16T22:00:22.477 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/udf/group.list 2023-12-16T22:00:22.491 INFO:teuthology.orchestra.run.smithi028.stdout:Building xfs 2023-12-16T22:00:22.498 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/xfs/group.list 2023-12-16T22:00:22.605 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo make install) 2023-12-16T22:00:22.630 INFO:teuthology.orchestra.run.smithi028.stdout:Building include 2023-12-16T22:00:22.634 INFO:teuthology.orchestra.run.smithi028.stdout:Building lib 2023-12-16T22:00:22.638 INFO:teuthology.orchestra.run.smithi028.stdout:Building ltp 2023-12-16T22:00:22.643 INFO:teuthology.orchestra.run.smithi028.stdout:Building src 2023-12-16T22:00:22.680 INFO:teuthology.orchestra.run.smithi028.stdout:Building log-writes 2023-12-16T22:00:22.684 INFO:teuthology.orchestra.run.smithi028.stdout:Building perf 2023-12-16T22:00:22.688 INFO:teuthology.orchestra.run.smithi028.stdout:Building aio-dio-regress 2023-12-16T22:00:22.692 INFO:teuthology.orchestra.run.smithi028.stdout:Building vfs 2023-12-16T22:00:22.696 INFO:teuthology.orchestra.run.smithi028.stdout:Building m4 2023-12-16T22:00:22.700 INFO:teuthology.orchestra.run.smithi028.stdout:Building common 2023-12-16T22:00:22.704 INFO:teuthology.orchestra.run.smithi028.stdout:Building tests 2023-12-16T22:00:22.710 INFO:teuthology.orchestra.run.smithi028.stdout:Building btrfs 2023-12-16T22:00:22.717 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/btrfs/group.list 2023-12-16T22:00:22.786 INFO:teuthology.orchestra.run.smithi028.stdout:Building ceph 2023-12-16T22:00:22.792 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ceph/group.list 2023-12-16T22:00:22.809 INFO:teuthology.orchestra.run.smithi028.stdout:Building cifs 2023-12-16T22:00:22.817 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/cifs/group.list 2023-12-16T22:00:22.829 INFO:teuthology.orchestra.run.smithi028.stdout:Building ext4 2023-12-16T22:00:22.835 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ext4/group.list 2023-12-16T22:00:22.881 INFO:teuthology.orchestra.run.smithi028.stdout:Building f2fs 2023-12-16T22:00:22.887 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/f2fs/group.list 2023-12-16T22:00:22.907 INFO:teuthology.orchestra.run.smithi028.stdout:Building generic 2023-12-16T22:00:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/generic/group.list 2023-12-16T22:00:23.025 INFO:teuthology.orchestra.run.smithi028.stdout:Building nfs 2023-12-16T22:00:23.031 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/nfs/group.list 2023-12-16T22:00:23.049 INFO:teuthology.orchestra.run.smithi028.stdout:Building ocfs2 2023-12-16T22:00:23.055 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ocfs2/group.list 2023-12-16T22:00:23.071 INFO:teuthology.orchestra.run.smithi028.stdout:Building overlay 2023-12-16T22:00:23.078 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/overlay/group.list 2023-12-16T22:00:23.125 INFO:teuthology.orchestra.run.smithi028.stdout:Building perf 2023-12-16T22:00:23.129 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/perf/group.list 2023-12-16T22:00:23.142 INFO:teuthology.orchestra.run.smithi028.stdout:Building selftest 2023-12-16T22:00:23.148 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/selftest/group.list 2023-12-16T22:00:23.161 INFO:teuthology.orchestra.run.smithi028.stdout:Building shared 2023-12-16T22:00:23.167 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/shared/group.list 2023-12-16T22:00:23.185 INFO:teuthology.orchestra.run.smithi028.stdout:Building tmpfs 2023-12-16T22:00:23.191 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/tmpfs/group.list 2023-12-16T22:00:23.207 INFO:teuthology.orchestra.run.smithi028.stdout:Building udf 2023-12-16T22:00:23.212 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/udf/group.list 2023-12-16T22:00:23.225 INFO:teuthology.orchestra.run.smithi028.stdout:Building xfs 2023-12-16T22:00:23.233 INFO:teuthology.orchestra.run.smithi028.stdout: [GROUP] /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/xfs/group.list 2023-12-16T22:00:23.339 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C include install 2023-12-16T22:00:23.342 INFO:teuthology.orchestra.run.smithi028.stdout:gmake[1]: Nothing to be done for 'install'. 2023-12-16T22:00:23.342 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C lib install 2023-12-16T22:00:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:gmake[1]: Nothing to be done for 'install'. 2023-12-16T22:00:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C ltp install 2023-12-16T22:00:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:../install-sh -o root -g root -m 755 -d /var/lib/xfstests/ltp 2023-12-16T22:00:23.359 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/bash ../libtool --quiet --mode=install ../install-sh -o root -g root -m 755 doio fsstress fsx growfiles iogen aio-stress /var/lib/xfstests/ltp 2023-12-16T22:00:23.449 INFO:teuthology.orchestra.run.smithi028.stdout:../install-sh -o root -g root -m 755 rwtest.sh /var/lib/xfstests/ltp 2023-12-16T22:00:23.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C src install 2023-12-16T22:00:23.481 INFO:teuthology.orchestra.run.smithi028.stdout:Building log-writes 2023-12-16T22:00:23.485 INFO:teuthology.orchestra.run.smithi028.stdout:Building perf 2023-12-16T22:00:23.489 INFO:teuthology.orchestra.run.smithi028.stdout:Building aio-dio-regress 2023-12-16T22:00:23.493 INFO:teuthology.orchestra.run.smithi028.stdout:Building vfs 2023-12-16T22:00:23.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake -C log-writes install 2023-12-16T22:00:23.499 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src/log-writes 2023-12-16T22:00:23.509 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 replay-log /var/lib/xfstests/src/log-writes 2023-12-16T22:00:23.520 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake -C perf install 2023-12-16T22:00:23.523 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src/perf 2023-12-16T22:00:23.534 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 FioCompare.py fio-insert-and-compare.py FioResultDecoder.py generate-schema.py ResultData.py fio-results.sql /var/lib/xfstests/src/perf 2023-12-16T22:00:23.559 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake -C aio-dio-regress install 2023-12-16T22:00:23.562 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src/aio-dio-regress 2023-12-16T22:00:23.573 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 aiocp aio-dio-append-write-fallocate-race aio-dio-append-write-read-race aio-dio-cow-race aio-dio-cycle-write aio-dio-eof-race aio-dio-extend-stat aio-dio-fcntl-race aio-dio-hole-filling-race aio-dio-invalidate-failure aio-dio-invalidate-readahead aiodio_sparse2 aio-dio-subblock-eof-read aio-dio-write-verify aio-free-ring-with-bogus-nr-pages aio-io-setup-with-nonwritable-context-pointer aio-last-ref-held-by-io /var/lib/xfstests/src/aio-dio-regress 2023-12-16T22:00:23.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake -C vfs install 2023-12-16T22:00:23.629 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src/vfs 2023-12-16T22:00:23.641 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 vfstest mount-idmapped /var/lib/xfstests/src/vfs 2023-12-16T22:00:23.654 INFO:teuthology.orchestra.run.smithi028.stdout:../install-sh -o root -g root -m 755 -d /var/lib/xfstests/src 2023-12-16T22:00:23.664 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/bash ../libtool --quiet --mode=install ../install-sh -o root -g root -m 755 dirstress fill fill2 getpagesize holes lstat64 nametest permname randholes runas truncfile usemem mmapcat append_reader append_writer dirperf metaperf devzero feature alloc fault fstest t_access_root godown resvtest writemod writev_on_pagefault makeextents itrash rename multi_open_unlink unwritten_sync genhashnames t_holes t_mmap_writev t_truncate_cmtime dirhash_collide t_rename_overwrite holetest t_truncate_self af_unix t_mmap_stale_pmd t_mmap_cow_race t_mmap_fallocate fsync-err t_mmap_write_ro t_ext4_dax_journal_corruption t_ext4_dax_inline_corruption t_ofd_locks t_mmap_collision mmap-write-concurrent t_get_file_time t_create_short_dirs t_create_long_dirs t_enospc t_mmap_writev_overlap checkpoint_journal mmap-rw-fault allocstale t_mmap_cow_memory_failure fake-dump-rootino dio-buf-fault rewinddir-test xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader preallo_rw_pattern_writer ftrunc trunc fs_perms testx looptest locktest unwritten_mmap bulkstat_unlink_test deduperace bulkstat_unlink_test_modified t_dir_offset t_futimens t_immutable stale_handle pwrite_mmap_blocked t_dir_offset2 seek_sanity_test seek_copy_test t_readdir_1 t_readdir_2 fsync-tester nsexec cloner renameat2 t_getcwd e4compact test-nextquota punch-alternating attr-list-by-handle-cursor-test listxattr dio-interleaved t_dir_type dio-invalidate-cache stat_test t_encrypted_d_revalidate attr_replace_test swapon mkswap t_attr_corruption t_open_tmpfiles fscrypt-crypt-util bulkstat_null_ocount splice-test chprojid_fail detached_mounts_propagation ext4_resize t_readdir_3 splice2pipe uuid_ioctl fiemap-tester t_stripealign open_by_handle t_mmap_dio uring_read_fault xfsfind t_reflink_read_race fssum /var/lib/xfstests/src 2023-12-16T22:00:25.306 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/bash ../libtool --quiet --mode=install ../install-sh -o root -g root -m 755 dmerror fill2attr fill2fs fill2fs_check scaleread.sh btrfs_crc32c_forged_name.py popdir.pl popattr.py soak_duration.awk /var/lib/xfstests/src 2023-12-16T22:00:25.437 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/bash ../libtool --quiet --mode=install ../install-sh -o root -g root -m 644 dumpfile /var/lib/xfstests/src 2023-12-16T22:00:25.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C m4 install 2023-12-16T22:00:25.472 INFO:teuthology.orchestra.run.smithi028.stdout:gmake[1]: Nothing to be done for 'install'. 2023-12-16T22:00:25.473 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C common install 2023-12-16T22:00:25.475 INFO:teuthology.orchestra.run.smithi028.stdout:../install-sh -o root -g root -m 755 -d /var/lib/xfstests/common 2023-12-16T22:00:25.486 INFO:teuthology.orchestra.run.smithi028.stdout:../install-sh -o root -g root -m 644 * /var/lib/xfstests/common 2023-12-16T22:00:25.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C tests install 2023-12-16T22:00:25.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/btrfs/ install 2023-12-16T22:00:25.643 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/btrfs 2023-12-16T22:00:25.654 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 /var/lib/xfstests/tests/btrfs 2023-12-16T22:00:26.643 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/btrfs 2023-12-16T22:00:26.656 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out 002.out 003.out 004.out 005.out 006.out 007.out 008.out 009.out 010.out 011.out 012.out 013.out 014.out 015.out 016.out 017.out 018.out 019.out 020.out 021.out 022.out 023.out 024.out 025.out 026.out 027.out 028.out 029.out 030.out 031.out 032.out 033.out 034.out 035.out 036.out 037.out 038.out 039.out 040.out 041.out 042.out 043.out 044.out 045.out 046.out 047.out 048.out 049.out 050.out 051.out 052.out 053.out 054.out 055.out 056.out 057.out 058.out 059.out 060.out 061.out 062.out 063.out 064.out 065.out 066.out 067.out 068.out 069.out 070.out 071.out 072.out 073.out 074.out 075.out 076.out 077.out 078.out 079.out 080.out 081.out 082.out 083.out 084.out 085.out 086.out 087.out 088.out 089.out 090.out 091.out 092.out 093.out 094.out 095.out 096.out 097.out 098.out 099.out 100.out 101.out 102.out 103.out 104.out 105.out 106.out 107.out 108.out 109.out 110.out 111.out 112.out 113.out 114.out 115.out 116.out 117.out 118.out 119.out 120.out 121.out 122.out 123.out 124.out 125.out 126.out 127.out 128.out 129.out 130.out 131.out 132.out 133.out 134.out 135.out 136.out 137.out 138.out 139.out 140.out 141.out 142.out 143.out 144.out 145.out 146.out 147.out 148.out 149.out 150.out 151.out 152.out 153.out 154.out 155.out 156.out 157.out 158.out 159.out 160.out 161.out 162.out 163.out 164.out 165.out 166.out 167.out 168.out 169.out 170.out 171.out 172.out 173.out 174.out 175.out 176.out 177.out 178.out 179.out 180.out 181.out 182.out 183.out 184.out 185.out 186.out 187.out 188.out 189.out 190.out 191.out 192.out 193.out 194.out 195.out 196.out 197.out 198.out 199.out 200.out 201.out 202.out 203.out 204.out 205.out 206.out 207.out 208.out 209.out 210.out 211.out 212.out 213.out 214.out 215.out 216.out 217.out 218.out 219.out 220.out 221.out 222.out 223.out 224.out 225.out 226.out 227.out 228.out 229.out 230.out 231.out 232.out 233.out 234.out 235.out 236.out 237.out 238.out 239.out 240.out 241.out 242.out 243.out 244.out 245.out 246.out 247.out 248.out 249.out 250.out 251.out 252.out 253.out 254.out 255.out 256.out 257.out 258.out 259.out 260.out 261.out 262.out 263.out 264.out 265.out 266.out 267.out 268.out 269.out 270.out 271.out 272.out 273.out 274.out 275.out 276.out 277.out 278.out 279.out 280.out 281.out 282.out 283.out 284.out 285.out 286.out 287.out 288.out 289.out 290.out 291.out 292.out 293.out 294.out 295.out 296.out 297.out 298.out 299.out 300.out 301.out 302.out 303.out /var/lib/xfstests/tests/btrfs 2023-12-16T22:00:27.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ceph/ install 2023-12-16T22:00:27.740 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/ceph 2023-12-16T22:00:27.754 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 /var/lib/xfstests/tests/ceph 2023-12-16T22:00:27.781 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/ceph 2023-12-16T22:00:27.793 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out 002.out 003.out 004.out 005.out /var/lib/xfstests/tests/ceph 2023-12-16T22:00:27.820 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/cifs/ install 2023-12-16T22:00:27.823 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/cifs 2023-12-16T22:00:27.836 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/cifs 2023-12-16T22:00:27.848 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/cifs 2023-12-16T22:00:27.861 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/cifs 2023-12-16T22:00:27.874 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ext4/ install 2023-12-16T22:00:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/ext4 2023-12-16T22:00:27.889 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 271 301 302 303 304 305 306 307 308 /var/lib/xfstests/tests/ext4 2023-12-16T22:00:28.130 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/ext4 2023-12-16T22:00:28.144 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out.default 001.out.nodelalloc 001.out.nozero 001.cfg 002.out 003.out 004.out 005.out 006.out 007.out 008.out 009.out 010.out 011.out 012.out 013.out 014.out 015.out 016.out 017.out 018.out 019.out 020.out 021.out 022.out 023.out 024.out 025.out 026.out 027.out 028.out 029.out 030.out 031.out 032.out 033.out 034.out 035.out 036.out 037.out 038.out 039.out 040.out 041.out 042.out 043.out 044.out 045.out 046.out 047.out 048.out 049.out 050.out 051.out 052.out 053.out 054.out 055.out 056.out 057.out 058.out 059.out 271.out 301.out 302.out 303.out 304.out 305.out 306.out 307.out 308.out /var/lib/xfstests/tests/ext4 2023-12-16T22:00:28.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/f2fs/ install 2023-12-16T22:00:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/f2fs 2023-12-16T22:00:28.419 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 002 /var/lib/xfstests/tests/f2fs 2023-12-16T22:00:28.435 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/f2fs 2023-12-16T22:00:28.448 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out 002.out /var/lib/xfstests/tests/f2fs 2023-12-16T22:00:28.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/generic/ install 2023-12-16T22:00:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/generic 2023-12-16T22:00:28.481 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 /var/lib/xfstests/tests/generic 2023-12-16T22:00:31.035 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/generic 2023-12-16T22:00:31.048 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 035.out.default 035.out.nfs 050.out.default 050.out.nojournal 050.out.xfsquota 035.cfg 050.cfg 001.out 002.out 003.out 004.out 005.out 006.out 007.out 008.out 009.out 010.out 011.out 012.out 013.out 014.out 015.out 016.out 017.out 018.out 019.out 020.out 021.out 022.out 023.out 024.out 025.out 026.out 027.out 028.out 029.out 030.out 031.out 032.out 033.out 034.out 036.out 037.out 038.out 039.out 040.out 041.out 042.out 043.out 044.out 045.out 046.out 047.out 048.out 049.out 051.out 052.out 053.out 054.out 055.out 056.out 057.out 058.out 059.out 060.out 061.out 062.out 063.out 064.out 065.out 066.out 067.out 068.out 069.out 070.out 071.out 072.out 073.out 074.out 075.out 076.out 077.out 078.out 079.out 080.out 081.out 082.out 083.out 084.out 085.out 086.out 087.out 088.out 089.out 090.out 091.out 092.out 093.out 094.out 095.out 096.out 097.out 098.out 099.out 100.out 101.out 102.out 103.out 104.out 105.out 106.out 107.out 108.out 109.out 110.out 111.out 112.out 113.out 114.out 115.out 116.out 117.out 118.out 119.out 120.out 121.out 122.out 123.out 124.out 125.out 126.out 127.out 128.out 129.out 130.out 131.out 132.out 133.out 134.out 135.out 136.out 137.out 138.out 139.out 140.out 141.out 142.out 143.out 144.out 145.out 146.out 147.out 148.out 149.out 150.out 151.out 152.out 153.out 154.out 155.out 156.out 157.out 158.out 159.out 160.out 161.out 162.out 163.out 164.out 165.out 166.out 167.out 168.out 169.out 170.out 171.out 172.out 173.out 174.out 175.out 176.out 177.out 178.out 179.out 180.out 181.out 182.out 183.out 184.out 185.out 186.out 187.out 188.out 189.out 190.out 191.out 192.out 193.out 194.out 195.out 196.out 197.out 198.out 199.out 200.out 201.out 202.out 203.out 204.out 205.out 206.out 207.out 208.out 209.out 210.out 211.out 212.out 213.out 214.out 215.out 216.out 217.out 218.out 219.out 220.out 221.out 222.out 223.out 224.out 225.out 226.out 227.out 228.out 229.out 230.out 231.out 232.out 233.out 234.out 235.out 236.out 237.out 238.out 239.out 240.out 241.out 242.out 243.out 244.out 245.out 246.out 247.out 248.out 249.out 250.out 251.out 252.out 253.out 254.out 255.out 256.out 257.out 258.out 259.out 260.out 261.out 262.out 263.out 264.out 265.out 266.out 267.out 268.out 269.out 270.out 271.out 272.out 273.out 274.out 275.out 276.out 277.out 278.out 279.out 280.out 281.out 282.out 283.out 284.out 285.out 286.out 287.out 288.out 289.out 290.out 291.out 292.out 293.out 294.out 295.out 296.out 297.out 298.out 299.out 300.out 301.out 302.out 303.out 304.out 305.out 306.out 307.out 308.out 309.out 310.out 311.out 312.out 313.out 314.out 315.out 316.out 317.out 318.out 319.out 320.out 321.out 322.out 323.out 324.out 325.out 326.out 327.out 328.out 329.out 330.out 331.out 332.out 333.out 334.out 335.out 336.out 337.out 338.out 339.out 340.out 341.out 342.out 343.out 344.out 345.out 346.out 347.out 348.out 349.out 350.out 351.out 352.out 353.out 354.out 355.out 356.out 357.out 358.out 359.out 360.out 361.out 362.out 363.out 364.out 365.out 366.out 367.out 368.out 369.out 370.out 371.out 372.out 373.out 374.out 375.out 376.out 377.out 378.out 379.out 380.out 381.out 382.out 383.out 384.out 385.out 386.out 387.out 388.out 389.out 390.out 391.out 392.out 393.out 394.out 395.out 396.out 397.out 398.out 399.out 400.out 401.out 402.out 403.out 404.out 405.out 406.out 407.out 408.out 409.out 410.out 411.out 412.out 413.out 414.out 415.out 416.out 417.out 418.out 419.out 420.out 421.out 422.out 423.out 424.out 425.out 426.out 427.out 428.out 429.out 430.out 431.out 432.out 433.out 434.out 435.out 436.out 437.out 438.out 439.out 440.out 441.out 442.out 443.out 444.out 445.out 446.out 447.out 448.out 449.out 450.out 451.out 452.out 453.out 454.out 455.out 456.out 457.out 458.out 459.out 460.out 461.out 462.out 463.out 464.out 465.out 466.out 467.out 468.out 469.out 470.out 471.out 472.out 473.out 474.out 475.out 476.out 477.out 478.out 479.out 480.out 481.out 482.out 483.out 484.out 485.out 486.out 487.out 488.out 489.out 490.out 491.out 492.out 493.out 494.out 495.out 496.out 497.out 498.out 499.out 500.out 501.out 502.out 503.out 504.out 505.out 506.out 507.out 508.out 509.out 510.out 511.out 512.out 513.out 514.out 515.out 516.out 517.out 518.out 519.out 520.out 521.out 522.out 523.out 524.out 525.out 526.out 527.out 528.out 529.out 530.out 531.out 532.out 533.out 534.out 535.out 536.out 537.out 538.out 539.out 540.out 541.out 542.out 543.out 544.out 545.out 546.out 547.out 548.out 549.out 550.out 551.out 552.out 553.out 554.out 555.out 556.out 557.out 558.out 559.out 560.out 561.out 562.out 563.out 564.out 565.out 566.out 567.out 568.out 569.out 570.out 571.out 572.out 573.out 574.out 575.out 576.out 577.out 578.out 579.out 580.out 581.out 582.out 583.out 584.out 585.out 586.out 587.out 588.out 589.out 590.out 591.out 592.out 593.out 594.out 595.out 596.out 597.out 598.out 599.out 600.out 601.out 602.out 603.out 604.out 605.out 606.out 607.out 608.out 609.out 610.out 611.out 612.out 613.out 614.out 615.out 616.out 617.out 618.out 619.out 620.out 621.out 622.out 623.out 624.out 625.out 626.out 627.out 628.out 629.out 630.out 631.out 632.out 633.out 634.out 635.out 636.out 637.out 638.out 639.out 640.out 641.out 642.out 643.out 644.out 645.out 646.out 647.out 648.out 649.out 650.out 651.out 652.out 653.out 654.out 655.out 656.out 657.out 658.out 659.out 660.out 661.out 662.out 663.out 664.out 665.out 666.out 667.out 668.out 669.out 670.out 671.out 672.out 673.out 674.out 675.out 676.out 677.out 678.out 679.out 680.out 681.out 682.out 683.out 684.out 685.out 686.out 687.out 688.out 689.out 690.out 691.out 692.out 693.out 694.out 695.out 696.out 697.out 698.out 699.out 700.out 701.out 702.out 703.out 704.out 705.out 706.out 707.out 708.out 709.out 710.out 711.out 712.out 713.out 714.out 715.out 716.out 717.out 718.out 719.out 720.out 721.out 722.out 723.out 724.out 725.out 726.out 727.out 728.out 729.out 730.out 731.out 732.out 733.out /var/lib/xfstests/tests/generic 2023-12-16T22:00:33.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/nfs/ install 2023-12-16T22:00:33.677 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/nfs 2023-12-16T22:00:33.690 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/nfs 2023-12-16T22:00:33.702 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/nfs 2023-12-16T22:00:33.714 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/nfs 2023-12-16T22:00:33.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/ocfs2/ install 2023-12-16T22:00:33.729 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/ocfs2 2023-12-16T22:00:33.742 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/ocfs2 2023-12-16T22:00:33.754 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/ocfs2 2023-12-16T22:00:33.766 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/ocfs2 2023-12-16T22:00:33.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/overlay/ install 2023-12-16T22:00:33.783 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/overlay 2023-12-16T22:00:33.795 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 /var/lib/xfstests/tests/overlay 2023-12-16T22:00:34.132 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/overlay 2023-12-16T22:00:34.144 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out 002.out 003.out 004.out 005.out 006.out 007.out 008.out 009.out 010.out 011.out 012.out 013.out 014.out 015.out 016.out 017.out 018.out 019.out 020.out 021.out 022.out 023.out 024.out 025.out 026.out 027.out 028.out 029.out 030.out 031.out 032.out 033.out 034.out 035.out 036.out 037.out 038.out 039.out 040.out 041.out 042.out 043.out 044.out 045.out 046.out 047.out 048.out 049.out 050.out 051.out 052.out 053.out 054.out 055.out 056.out 057.out 058.out 059.out 060.out 061.out 062.out 063.out 064.out 065.out 066.out 067.out 068.out 069.out 070.out 071.out 072.out 073.out 074.out 075.out 076.out 077.out 078.out 079.out 080.out 081.out 082.out 083.out 100.out 101.out 102.out 103.out 104.out 105.out 106.out 107.out 108.out 109.out 110.out 111.out 112.out 113.out 114.out 115.out 116.out 117.out /var/lib/xfstests/tests/overlay 2023-12-16T22:00:34.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/perf/ install 2023-12-16T22:00:34.501 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/perf 2023-12-16T22:00:34.514 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/perf 2023-12-16T22:00:34.527 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/perf 2023-12-16T22:00:34.539 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/perf 2023-12-16T22:00:34.552 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/selftest/ install 2023-12-16T22:00:34.555 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/selftest 2023-12-16T22:00:34.568 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 /var/lib/xfstests/tests/selftest 2023-12-16T22:00:34.596 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/selftest 2023-12-16T22:00:34.609 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out 002.out 003.out 004.out 005.out 006.out /var/lib/xfstests/tests/selftest 2023-12-16T22:00:34.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/shared/ install 2023-12-16T22:00:34.642 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/shared 2023-12-16T22:00:34.655 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 002 032 298 /var/lib/xfstests/tests/shared 2023-12-16T22:00:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/shared 2023-12-16T22:00:34.686 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 002.out 032.out 298.out /var/lib/xfstests/tests/shared 2023-12-16T22:00:34.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/tmpfs/ install 2023-12-16T22:00:34.708 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/tmpfs 2023-12-16T22:00:34.720 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 /var/lib/xfstests/tests/tmpfs 2023-12-16T22:00:34.732 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/tmpfs 2023-12-16T22:00:34.744 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 001.out /var/lib/xfstests/tests/tmpfs 2023-12-16T22:00:34.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/udf/ install 2023-12-16T22:00:34.759 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/udf 2023-12-16T22:00:34.770 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 102 /var/lib/xfstests/tests/udf 2023-12-16T22:00:34.783 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/udf 2023-12-16T22:00:34.795 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 102.out /var/lib/xfstests/tests/udf 2023-12-16T22:00:34.807 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/gmake --no-print-directory Q=@ -C /tmp/tmp.nC0MjoAvrDxfstests-dev/tests/xfs/ install 2023-12-16T22:00:34.812 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests/xfs 2023-12-16T22:00:34.824 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 755 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 /var/lib/xfstests/tests/xfs 2023-12-16T22:00:37.231 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 group.list /var/lib/xfstests/tests/xfs 2023-12-16T22:00:37.244 INFO:teuthology.orchestra.run.smithi028.stdout:../../install-sh -o root -g root -m 644 033.out.crc 033.out.default 071.out.32 071.out.64 096.out.external 096.out.internal 216.out.64mblog 216.out.classic 033.cfg 216.cfg 001.out 002.out 003.out 004.out 005.out 006.out 007.out 008.out 009.out 010.out 011.out 012.out 013.out 014.out 015.out 016.out 017.out 018.out 019.out 020.out 021.out 022.out 023.out 024.out 025.out 026.out 027.out 028.out 029.out 030.out 031.out 032.out 034.out 035.out 036.out 037.out 038.out 039.out 040.out 041.out 042.out 043.out 044.out 045.out 046.out 047.out 048.out 049.out 050.out 051.out 052.out 053.out 054.out 055.out 056.out 057.out 058.out 059.out 060.out 061.out 062.out 063.out 064.out 065.out 066.out 067.out 068.out 069.out 070.out 072.out 073.out 074.out 075.out 076.out 077.out 078.out 079.out 080.out 081.out 082.out 083.out 084.out 085.out 086.out 087.out 088.out 089.out 090.out 091.out 092.out 093.out 094.out 095.out 097.out 098.out 099.out 100.out 101.out 102.out 103.out 104.out 105.out 106.out 107.out 108.out 109.out 110.out 111.out 112.out 113.out 114.out 115.out 116.out 117.out 118.out 119.out 120.out 121.out 122.out 123.out 124.out 125.out 126.out 127.out 128.out 129.out 130.out 131.out 132.out 133.out 134.out 135.out 136.out 137.out 138.out 139.out 140.out 141.out 142.out 143.out 144.out 145.out 146.out 147.out 148.out 149.out 150.out 151.out 152.out 153.out 154.out 155.out 156.out 157.out 158.out 159.out 160.out 161.out 162.out 163.out 164.out 165.out 166.out 167.out 168.out 169.out 170.out 171.out 172.out 173.out 174.out 175.out 176.out 177.out 178.out 179.out 180.out 181.out 182.out 183.out 184.out 185.out 186.out 187.out 188.out 189.out 190.out 191.out 192.out 193.out 194.out 195.out 196.out 197.out 198.out 199.out 200.out 201.out 202.out 203.out 204.out 205.out 206.out 207.out 208.out 209.out 210.out 211.out 212.out 213.out 214.out 215.out 217.out 218.out 219.out 220.out 221.out 222.out 223.out 224.out 225.out 226.out 227.out 228.out 229.out 230.out 231.out 232.out 233.out 234.out 235.out 236.out 237.out 238.out 239.out 240.out 241.out 242.out 243.out 244.out 245.out 246.out 247.out 248.out 249.out 250.out 251.out 252.out 253.out 254.out 255.out 256.out 257.out 258.out 259.out 260.out 261.out 262.out 263.out 264.out 265.out 266.out 267.out 268.out 269.out 270.out 271.out 272.out 273.out 274.out 275.out 276.out 277.out 278.out 279.out 280.out 281.out 282.out 283.out 284.out 285.out 286.out 287.out 288.out 289.out 290.out 291.out 292.out 293.out 294.out 295.out 296.out 297.out 298.out 299.out 300.out 301.out 302.out 303.out 304.out 305.out 306.out 307.out 308.out 309.out 310.out 311.out 312.out 313.out 314.out 315.out 316.out 317.out 318.out 319.out 320.out 321.out 322.out 323.out 324.out 325.out 326.out 327.out 328.out 329.out 330.out 331.out 332.out 333.out 334.out 335.out 336.out 337.out 338.out 339.out 340.out 341.out 342.out 343.out 344.out 345.out 346.out 347.out 348.out 349.out 350.out 351.out 352.out 353.out 354.out 355.out 356.out 357.out 358.out 359.out 360.out 361.out 362.out 363.out 364.out 365.out 366.out 367.out 368.out 369.out 370.out 371.out 372.out 373.out 374.out 375.out 376.out 377.out 378.out 379.out 380.out 381.out 382.out 383.out 384.out 385.out 386.out 387.out 388.out 389.out 390.out 391.out 392.out 393.out 394.out 395.out 396.out 397.out 398.out 399.out 400.out 401.out 402.out 403.out 404.out 405.out 406.out 407.out 408.out 409.out 410.out 411.out 412.out 413.out 414.out 415.out 416.out 417.out 418.out 419.out 420.out 421.out 422.out 423.out 424.out 425.out 426.out 427.out 428.out 429.out 430.out 431.out 432.out 433.out 434.out 435.out 436.out 437.out 438.out 439.out 440.out 441.out 442.out 443.out 444.out 445.out 446.out 447.out 448.out 449.out 450.out 451.out 452.out 453.out 454.out 455.out 456.out 457.out 458.out 459.out 460.out 461.out 462.out 463.out 464.out 465.out 466.out 467.out 468.out 469.out 470.out 471.out 472.out 473.out 474.out 475.out 476.out 477.out 478.out 479.out 480.out 481.out 482.out 483.out 484.out 485.out 486.out 487.out 488.out 489.out 490.out 491.out 492.out 493.out 494.out 495.out 496.out 497.out 498.out 499.out 500.out 501.out 502.out 503.out 504.out 505.out 506.out 507.out 508.out 509.out 510.out 511.out 512.out 513.out 514.out 515.out 516.out 517.out 518.out 519.out 520.out 521.out 522.out 523.out 524.out 525.out 526.out 527.out 528.out 529.out 530.out 531.out 532.out 533.out 534.out 535.out 536.out 537.out 538.out 539.out 540.out 541.out 542.out 543.out 544.out 545.out 546.out 547.out 548.out 549.out 550.out 551.out 552.out 553.out 554.out 555.out 556.out 557.out 558.out 559.out 560.out 561.out 562.out 563.out 564.out 565.out 566.out 567.out 568.out 569.out 570.out 571.out 572.out 573.out 574.out 575.out 576.out 577.out 578.out 579.out 580.out 581.out 582.out 583.out 584.out 585.out 586.out 587.out 588.out 589.out 590.out 591.out 592.out 593.out 594.out 595.out 596.out 597.out 598.out 599.out 600.out 601.out 602.out 603.out 604.out 708.out 709.out 710.out 711.out 712.out 713.out 714.out 715.out 716.out 717.out 718.out 719.out 720.out 721.out 722.out 723.out 724.out 725.out 726.out 727.out 728.out 729.out 730.out 731.out 732.out 733.out 734.out 735.out 736.out 737.out 738.out 739.out 740.out 741.out 742.out 743.out 744.out 745.out 746.out 747.out 748.out 749.out 750.out 751.out 752.out 753.out 754.out 755.out 756.out 757.out 758.out 759.out 760.out 761.out 762.out 763.out 764.out 765.out 766.out 767.out 768.out 769.out 770.out 771.out 772.out 773.out 774.out 775.out 776.out 777.out 778.out 779.out 780.out 781.out 782.out 783.out 784.out 785.out 786.out 787.out 788.out 789.out 790.out 791.out 792.out 793.out 794.out 795.out 796.out 797.out 798.out 799.out 800.out /var/lib/xfstests/tests/xfs 2023-12-16T22:00:39.716 INFO:teuthology.orchestra.run.smithi028.stdout:../install-sh -o root -g root -m 755 -d /var/lib/xfstests/tests 2023-12-16T22:00:39.729 INFO:teuthology.orchestra.run.smithi028.stdout:./install-sh -o root -g root -m 755 -d /var/lib/xfstests 2023-12-16T22:00:39.740 INFO:teuthology.orchestra.run.smithi028.stdout:./install-sh -o root -g root -m 755 check /var/lib/xfstests 2023-12-16T22:00:39.753 INFO:teuthology.orchestra.run.smithi028.stdout:./install-sh -o root -g root -m 644 randomize.awk /var/lib/xfstests 2023-12-16T22:00:39.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T22:00:39.769 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec ls tests/generic) 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:001 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:001.out 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:002 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:002.out 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:003 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:003.out 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:004 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:004.out 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:005 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:005.out 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:006 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:006.out 2023-12-16T22:00:39.819 INFO:teuthology.orchestra.run.smithi028.stdout:007 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:007.out 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:008 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:008.out 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:009 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:009.out 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:010 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:010.out 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:011 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:011.out 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:012 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:012.out 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:013 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:013.out 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:014 2023-12-16T22:00:39.820 INFO:teuthology.orchestra.run.smithi028.stdout:014.out 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:015 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:015.out 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:016 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:016.out 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:017 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:017.out 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:018 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:018.out 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:019 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:019.out 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:020 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:020.out 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:021 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:021.out 2023-12-16T22:00:39.821 INFO:teuthology.orchestra.run.smithi028.stdout:022 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:022.out 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:023 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:023.out 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:024 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:024.out 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:025 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:025.out 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:026 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:026.out 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:027 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:027.out 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:028 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:028.out 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:029 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:029.out 2023-12-16T22:00:39.822 INFO:teuthology.orchestra.run.smithi028.stdout:030 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:030.out 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:031 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:031.out 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:032 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:032.out 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:033 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:033.out 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:034 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:034.out 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:035 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:035.cfg 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:035.out.default 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:035.out.nfs 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:036 2023-12-16T22:00:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:036.out 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:037 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:037.out 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:038 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:038.out 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:039 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:039.out 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:040 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:040.out 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:041 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:041.out 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:042 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:042.out 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:043 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:043.out 2023-12-16T22:00:39.824 INFO:teuthology.orchestra.run.smithi028.stdout:044 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:044.out 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:045 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:045.out 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:046 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:046.out 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:047 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:047.out 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:048 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:048.out 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:049 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:049.out 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:050 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:050.cfg 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:050.out.default 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:050.out.nojournal 2023-12-16T22:00:39.825 INFO:teuthology.orchestra.run.smithi028.stdout:050.out.xfsquota 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:051 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:051.out 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:052 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:052.out 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:053 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:053.out 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:054 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:054.out 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:055 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:055.out 2023-12-16T22:00:39.826 INFO:teuthology.orchestra.run.smithi028.stdout:056 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:056.out 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:057 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:057.out 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:058 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:058.out 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:059 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:059.out 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:060 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:060.out 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:061 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:061.out 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:062 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:062.out 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:063 2023-12-16T22:00:39.827 INFO:teuthology.orchestra.run.smithi028.stdout:063.out 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:064 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:064.out 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:065 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:065.out 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:066 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:066.out 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:067 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:067.out 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:068 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:068.out 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:069 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:069.out 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:070 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:070.out 2023-12-16T22:00:39.828 INFO:teuthology.orchestra.run.smithi028.stdout:071 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:071.out 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:072 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:072.out 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:073 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:073.out 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:074 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:074.out 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:075 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:075.out 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:076 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:076.out 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:077 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:077.out 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:078 2023-12-16T22:00:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:078.out 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:079 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:079.out 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:080 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:080.out 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:081 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:081.out 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:082 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:082.out 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:083 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:083.out 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:084 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:084.out 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:085 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:085.out 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:086 2023-12-16T22:00:39.830 INFO:teuthology.orchestra.run.smithi028.stdout:086.out 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:087 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:087.out 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:088 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:088.out 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:089 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:089.out 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:090 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:090.out 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:091 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:091.out 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:092 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:092.out 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:093 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:093.out 2023-12-16T22:00:39.831 INFO:teuthology.orchestra.run.smithi028.stdout:094 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:094.out 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:095 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:095.out 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:096 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:096.out 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:097 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:097.out 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:098 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:098.out 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:099 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:099.out 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:100 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:100.out 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:101 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:101.out 2023-12-16T22:00:39.832 INFO:teuthology.orchestra.run.smithi028.stdout:102 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:102.out 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:103 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:103.out 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:104 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:104.out 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:105 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:105.out 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:106 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:106.out 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:107 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:107.out 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:108 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:108.out 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:109 2023-12-16T22:00:39.833 INFO:teuthology.orchestra.run.smithi028.stdout:109.out 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:110 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:110.out 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:111 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:111.out 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:112 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:112.out 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:113 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:113.out 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:114 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:114.out 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:115 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:115.out 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:116 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:116.out 2023-12-16T22:00:39.834 INFO:teuthology.orchestra.run.smithi028.stdout:117 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:117.out 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:118 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:118.out 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:119 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:119.out 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:120 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:120.out 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:121 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:121.out 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:122 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:122.out 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:123 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:123.out 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:124 2023-12-16T22:00:39.835 INFO:teuthology.orchestra.run.smithi028.stdout:124.out 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:125 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:125.out 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:126 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:126.out 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:127 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:127.out 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:128 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:128.out 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:129 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:129.out 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:130 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:130.out 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:131 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:131.out 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:132 2023-12-16T22:00:39.836 INFO:teuthology.orchestra.run.smithi028.stdout:132.out 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:133 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:133.out 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:134 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:134.out 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:135 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:135.out 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:136 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:136.out 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:137 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:137.out 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:138 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:138.out 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:139 2023-12-16T22:00:39.837 INFO:teuthology.orchestra.run.smithi028.stdout:139.out 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:140 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:140.out 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:141 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:141.out 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:142 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:142.out 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:143 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:143.out 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:144 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:144.out 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:145 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:145.out 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:146 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:146.out 2023-12-16T22:00:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:147 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:147.out 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:148 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:148.out 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:149 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:149.out 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:150 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:150.out 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:151 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:151.out 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:152 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:152.out 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:153 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:153.out 2023-12-16T22:00:39.839 INFO:teuthology.orchestra.run.smithi028.stdout:154 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:154.out 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:155 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:155.out 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:156 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:156.out 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:157 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:157.out 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:158 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:158.out 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:159 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:159.out 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:160 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:160.out 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:161 2023-12-16T22:00:39.840 INFO:teuthology.orchestra.run.smithi028.stdout:161.out 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:162 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:162.out 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:163 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:163.out 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:164 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:164.out 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:165 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:165.out 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:166 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:166.out 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:167 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:167.out 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:168 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:168.out 2023-12-16T22:00:39.841 INFO:teuthology.orchestra.run.smithi028.stdout:169 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:169.out 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:170 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:170.out 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:171 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:171.out 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:172 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:172.out 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:173 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:173.out 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:174 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:174.out 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:175 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:175.out 2023-12-16T22:00:39.842 INFO:teuthology.orchestra.run.smithi028.stdout:176 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:176.out 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:177 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:177.out 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:178 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:178.out 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:179 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:179.out 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:180 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:180.out 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:181 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:181.out 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:182 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:182.out 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:183 2023-12-16T22:00:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:183.out 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:184 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:184.out 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:185 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:185.out 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:186 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:186.out 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:187 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:187.out 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:188 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:188.out 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:189 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:189.out 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:190 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:190.out 2023-12-16T22:00:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:191 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:191.out 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:192 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:192.out 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:193 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:193.out 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:194 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:194.out 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:195 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:195.out 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:196 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:196.out 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:197 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:197.out 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:198 2023-12-16T22:00:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:198.out 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:199 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:199.out 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:200 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:200.out 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:201 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:201.out 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:202 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:202.out 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:203 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:203.out 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:204 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:204.out 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:205 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:205.out 2023-12-16T22:00:39.846 INFO:teuthology.orchestra.run.smithi028.stdout:206 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:206.out 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:207 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:207.out 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:208 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:208.out 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:209 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:209.out 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:210 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:210.out 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:211 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:211.out 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:212 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:212.out 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:213 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:213.out 2023-12-16T22:00:39.847 INFO:teuthology.orchestra.run.smithi028.stdout:214 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:214.out 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:215 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:215.out 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:216 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:216.out 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:217 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:217.out 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:218 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:218.out 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:219 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:219.out 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:220 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:220.out 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:221 2023-12-16T22:00:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:221.out 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:222 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:222.out 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:223 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:223.out 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:224 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:224.out 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:225 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:225.out 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:226 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:226.out 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:227 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:227.out 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:228 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:228.out 2023-12-16T22:00:39.849 INFO:teuthology.orchestra.run.smithi028.stdout:229 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:229.out 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:230 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:230.out 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:231 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:231.out 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:232 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:232.out 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:233 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:233.out 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:234 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:234.out 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:235 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:235.out 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:236 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:236.out 2023-12-16T22:00:39.850 INFO:teuthology.orchestra.run.smithi028.stdout:237 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:237.out 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:238 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:238.out 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:239 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:239.out 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:240 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:240.out 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:241 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:241.out 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:242 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:242.out 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:243 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:243.out 2023-12-16T22:00:39.851 INFO:teuthology.orchestra.run.smithi028.stdout:244 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:244.out 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:245 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:245.out 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:246 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:246.out 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:247 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:247.out 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:248 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:248.out 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:249 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:249.out 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:250 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:250.out 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:251 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:251.out 2023-12-16T22:00:39.852 INFO:teuthology.orchestra.run.smithi028.stdout:252 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:252.out 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:253 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:253.out 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:254 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:254.out 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:255 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:255.out 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:256 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:256.out 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:257 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:257.out 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:258 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:258.out 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:259 2023-12-16T22:00:39.853 INFO:teuthology.orchestra.run.smithi028.stdout:259.out 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:260 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:260.out 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:261 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:261.out 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:262 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:262.out 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:263 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:263.out 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:264 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:264.out 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:265 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:265.out 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:266 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:266.out 2023-12-16T22:00:39.854 INFO:teuthology.orchestra.run.smithi028.stdout:267 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:267.out 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:268 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:268.out 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:269 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:269.out 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:270 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:270.out 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:271 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:271.out 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:272 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:272.out 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:273 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:273.out 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:274 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:274.out 2023-12-16T22:00:39.855 INFO:teuthology.orchestra.run.smithi028.stdout:275 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:275.out 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:276 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:276.out 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:277 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:277.out 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:278 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:278.out 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:279 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:279.out 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:280 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:280.out 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:281 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:281.out 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:282 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:282.out 2023-12-16T22:00:39.856 INFO:teuthology.orchestra.run.smithi028.stdout:283 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:283.out 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:284 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:284.out 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:285 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:285.out 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:286 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:286.out 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:287 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:287.out 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:288 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:288.out 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:289 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:289.out 2023-12-16T22:00:39.857 INFO:teuthology.orchestra.run.smithi028.stdout:290 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:290.out 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:291 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:291.out 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:292 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:292.out 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:293 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:293.out 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:294 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:294.out 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:295 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:295.out 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:296 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:296.out 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:297 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:297.out 2023-12-16T22:00:39.858 INFO:teuthology.orchestra.run.smithi028.stdout:298 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:298.out 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:299 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:299.out 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:300 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:300.out 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:301 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:301.out 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:302 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:302.out 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:303 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:303.out 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:304 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:304.out 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:305 2023-12-16T22:00:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:305.out 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:306 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:306.out 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:307 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:307.out 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:308 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:308.out 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:309 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:309.out 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:310 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:310.out 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:311 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:311.out 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:312 2023-12-16T22:00:39.860 INFO:teuthology.orchestra.run.smithi028.stdout:312.out 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:313 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:313.out 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:314 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:314.out 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:315 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:315.out 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:316 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:316.out 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:317 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:317.out 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:318 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:318.out 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:319 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:319.out 2023-12-16T22:00:39.861 INFO:teuthology.orchestra.run.smithi028.stdout:320 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:320.out 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:321 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:321.out 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:322 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:322.out 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:323 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:323.out 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:324 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:324.out 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:325 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:325.out 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:326 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:326.out 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:327 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:327.out 2023-12-16T22:00:39.862 INFO:teuthology.orchestra.run.smithi028.stdout:328 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:328.out 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:329 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:329.out 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:330 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:330.out 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:331 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:331.out 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:332 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:332.out 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:333 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:333.out 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:334 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:334.out 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:335 2023-12-16T22:00:39.863 INFO:teuthology.orchestra.run.smithi028.stdout:335.out 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:336 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:336.out 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:337 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:337.out 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:338 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:338.out 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:339 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:339.out 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:340 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:340.out 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:341 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:341.out 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:342 2023-12-16T22:00:39.864 INFO:teuthology.orchestra.run.smithi028.stdout:342.out 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:343 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:343.out 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:344 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:344.out 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:345 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:345.out 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:346 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:346.out 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:347 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:347.out 2023-12-16T22:00:39.865 INFO:teuthology.orchestra.run.smithi028.stdout:348 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:348.out 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:349 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:349.out 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:350 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:350.out 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:351 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:351.out 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:352 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:352.out 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:353 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:353.out 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:354 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:354.out 2023-12-16T22:00:39.866 INFO:teuthology.orchestra.run.smithi028.stdout:355 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:355.out 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:356 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:356.out 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:357 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:357.out 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:358 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:358.out 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:359 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:359.out 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:360 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:360.out 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:361 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:361.out 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:362 2023-12-16T22:00:39.867 INFO:teuthology.orchestra.run.smithi028.stdout:362.out 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:363 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:363.out 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:364 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:364.out 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:365 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:365.out 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:366 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:366.out 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:367 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:367.out 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:368 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:368.out 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:369 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:369.out 2023-12-16T22:00:39.868 INFO:teuthology.orchestra.run.smithi028.stdout:370 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:370.out 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:371 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:371.out 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:372 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:372.out 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:373 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:373.out 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:374 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:374.out 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:375 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:375.out 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:376 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:376.out 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:377 2023-12-16T22:00:39.869 INFO:teuthology.orchestra.run.smithi028.stdout:377.out 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:378 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:378.out 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:379 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:379.out 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:380 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:380.out 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:381 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:381.out 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:382 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:382.out 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:383 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:383.out 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:384 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:384.out 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:385 2023-12-16T22:00:39.870 INFO:teuthology.orchestra.run.smithi028.stdout:385.out 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:386 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:386.out 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:387 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:387.out 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:388 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:388.out 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:389 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:389.out 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:390 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:390.out 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:391 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:391.out 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:392 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:392.out 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:393 2023-12-16T22:00:39.871 INFO:teuthology.orchestra.run.smithi028.stdout:393.out 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:394 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:394.out 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:395 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:395.out 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:396 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:396.out 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:397 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:397.out 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:398 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:398.out 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:399 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:399.out 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:400 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:400.out 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:401 2023-12-16T22:00:39.872 INFO:teuthology.orchestra.run.smithi028.stdout:401.out 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:402 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:402.out 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:403 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:403.out 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:404 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:404.out 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:405 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:405.out 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:406 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:406.out 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:407 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:407.out 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:408 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:408.out 2023-12-16T22:00:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:409 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:409.out 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:410 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:410.out 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:411 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:411.out 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:412 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:412.out 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:413 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:413.out 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:414 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:414.out 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:415 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:415.out 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:416 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:416.out 2023-12-16T22:00:39.874 INFO:teuthology.orchestra.run.smithi028.stdout:417 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:417.out 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:418 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:418.out 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:419 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:419.out 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:420 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:420.out 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:421 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:421.out 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:422 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:422.out 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:423 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:423.out 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:424 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:424.out 2023-12-16T22:00:39.875 INFO:teuthology.orchestra.run.smithi028.stdout:425 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:425.out 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:426 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:426.out 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:427 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:427.out 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:428 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:428.out 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:429 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:429.out 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:430 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:430.out 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:431 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:431.out 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:432 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:432.out 2023-12-16T22:00:39.876 INFO:teuthology.orchestra.run.smithi028.stdout:433 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:433.out 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:434 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:434.out 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:435 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:435.out 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:436 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:436.out 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:437 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:437.out 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:438 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:438.out 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:439 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:439.out 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:440 2023-12-16T22:00:39.877 INFO:teuthology.orchestra.run.smithi028.stdout:440.out 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:441 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:441.out 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:442 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:442.out 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:443 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:443.out 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:444 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:444.out 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:445 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:445.out 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:446 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:446.out 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:447 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:447.out 2023-12-16T22:00:39.878 INFO:teuthology.orchestra.run.smithi028.stdout:448 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:448.out 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:449 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:449.out 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:450 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:450.out 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:451 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:451.out 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:452 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:452.out 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:453 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:453.out 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:454 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:454.out 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:455 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:455.out 2023-12-16T22:00:39.879 INFO:teuthology.orchestra.run.smithi028.stdout:456 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:456.out 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:457 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:457.out 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:458 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:458.out 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:459 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:459.out 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:460 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:460.out 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:461 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:461.out 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:462 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:462.out 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:463 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:463.out 2023-12-16T22:00:39.880 INFO:teuthology.orchestra.run.smithi028.stdout:464 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:464.out 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:465 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:465.out 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:466 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:466.out 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:467 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:467.out 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:468 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:468.out 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:469 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:469.out 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:470 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:470.out 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:471 2023-12-16T22:00:39.881 INFO:teuthology.orchestra.run.smithi028.stdout:471.out 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:472 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:472.out 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:473 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:473.out 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:474 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:474.out 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:475 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:475.out 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:476 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:476.out 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:477 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:477.out 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:478 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:478.out 2023-12-16T22:00:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:479 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:479.out 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:480 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:480.out 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:481 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:481.out 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:482 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:482.out 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:483 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:483.out 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:484 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:484.out 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:485 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:485.out 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:486 2023-12-16T22:00:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:486.out 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:487 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:487.out 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:488 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:488.out 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:489 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:489.out 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:490 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:490.out 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:491 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:491.out 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:492 2023-12-16T22:00:39.884 INFO:teuthology.orchestra.run.smithi028.stdout:492.out 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:493 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:493.out 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:494 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:494.out 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:495 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:495.out 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:496 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:496.out 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:497 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:497.out 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:498 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:498.out 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:499 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:499.out 2023-12-16T22:00:39.885 INFO:teuthology.orchestra.run.smithi028.stdout:500 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:500.out 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:501 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:501.out 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:502 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:502.out 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:503 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:503.out 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:504 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:504.out 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:505 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:505.out 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:506 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:506.out 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:507 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:507.out 2023-12-16T22:00:39.886 INFO:teuthology.orchestra.run.smithi028.stdout:508 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:508.out 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:509 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:509.out 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:510 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:510.out 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:511 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:511.out 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:512 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:512.out 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:513 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:513.out 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:514 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:514.out 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:515 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:515.out 2023-12-16T22:00:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:516 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:516.out 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:517 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:517.out 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:518 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:518.out 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:519 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:519.out 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:520 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:520.out 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:521 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:521.out 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:522 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:522.out 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:523 2023-12-16T22:00:39.888 INFO:teuthology.orchestra.run.smithi028.stdout:523.out 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:524 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:524.out 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:525 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:525.out 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:526 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:526.out 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:527 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:527.out 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:528 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:528.out 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:529 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:529.out 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:530 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:530.out 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:531 2023-12-16T22:00:39.889 INFO:teuthology.orchestra.run.smithi028.stdout:531.out 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:532 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:532.out 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:533 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:533.out 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:534 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:534.out 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:535 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:535.out 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:536 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:536.out 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:537 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:537.out 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:538 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:538.out 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:539 2023-12-16T22:00:39.890 INFO:teuthology.orchestra.run.smithi028.stdout:539.out 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:540 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:540.out 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:541 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:541.out 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:542 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:542.out 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:543 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:543.out 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:544 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:544.out 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:545 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:545.out 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:546 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:546.out 2023-12-16T22:00:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:547 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:547.out 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:548 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:548.out 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:549 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:549.out 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:550 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:550.out 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:551 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:551.out 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:552 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:552.out 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:553 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:553.out 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:554 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:554.out 2023-12-16T22:00:39.892 INFO:teuthology.orchestra.run.smithi028.stdout:555 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:555.out 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:556 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:556.out 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:557 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:557.out 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:558 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:558.out 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:559 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:559.out 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:560 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:560.out 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:561 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:561.out 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:562 2023-12-16T22:00:39.893 INFO:teuthology.orchestra.run.smithi028.stdout:562.out 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:563 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:563.out 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:564 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:564.out 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:565 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:565.out 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:566 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:566.out 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:567 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:567.out 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:568 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:568.out 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:569 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:569.out 2023-12-16T22:00:39.894 INFO:teuthology.orchestra.run.smithi028.stdout:570 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:570.out 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:571 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:571.out 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:572 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:572.out 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:573 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:573.out 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:574 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:574.out 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:575 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:575.out 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:576 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:576.out 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:577 2023-12-16T22:00:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:577.out 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:578 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:578.out 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:579 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:579.out 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:580 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:580.out 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:581 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:581.out 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:582 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:582.out 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:583 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:583.out 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:584 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:584.out 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:585 2023-12-16T22:00:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:585.out 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:586 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:586.out 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:587 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:587.out 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:588 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:588.out 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:589 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:589.out 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:590 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:590.out 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:591 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:591.out 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:592 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:592.out 2023-12-16T22:00:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:593 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:593.out 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:594 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:594.out 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:595 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:595.out 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:596 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:596.out 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:597 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:597.out 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:598 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:598.out 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:599 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:599.out 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:600 2023-12-16T22:00:39.898 INFO:teuthology.orchestra.run.smithi028.stdout:600.out 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:601 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:601.out 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:602 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:602.out 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:603 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:603.out 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:604 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:604.out 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:605 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:605.out 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:606 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:606.out 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:607 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:607.out 2023-12-16T22:00:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:608 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:608.out 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:609 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:609.out 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:610 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:610.out 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:611 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:611.out 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:612 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:612.out 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:613 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:613.out 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:614 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:614.out 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:615 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:615.out 2023-12-16T22:00:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:616 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:616.out 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:617 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:617.out 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:618 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:618.out 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:619 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:619.out 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:620 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:620.out 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:621 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:621.out 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:622 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:622.out 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:623 2023-12-16T22:00:39.901 INFO:teuthology.orchestra.run.smithi028.stdout:623.out 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:624 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:624.out 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:625 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:625.out 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:626 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:626.out 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:627 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:627.out 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:628 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:628.out 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:629 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:629.out 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:630 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:630.out 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:631 2023-12-16T22:00:39.902 INFO:teuthology.orchestra.run.smithi028.stdout:631.out 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:632 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:632.out 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:633 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:633.out 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:634 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:634.out 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:635 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:635.out 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:636 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:636.out 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:637 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:637.out 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:638 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:638.out 2023-12-16T22:00:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:639 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:639.out 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:640 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:640.out 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:641 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:641.out 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:642 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:642.out 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:643 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:643.out 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:644 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:644.out 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:645 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:645.out 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:646 2023-12-16T22:00:39.904 INFO:teuthology.orchestra.run.smithi028.stdout:646.out 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:647 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:647.out 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:648 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:648.out 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:649 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:649.out 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:650 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:650.out 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:651 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:651.out 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:652 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:652.out 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:653 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:653.out 2023-12-16T22:00:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:654 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:654.out 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:655 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:655.out 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:656 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:656.out 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:657 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:657.out 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:658 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:658.out 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:659 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:659.out 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:660 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:660.out 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:661 2023-12-16T22:00:39.906 INFO:teuthology.orchestra.run.smithi028.stdout:661.out 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:662 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:662.out 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:663 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:663.out 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:664 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:664.out 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:665 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:665.out 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:666 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:666.out 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:667 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:667.out 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:668 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:668.out 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:669 2023-12-16T22:00:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:669.out 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:670 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:670.out 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:671 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:671.out 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:672 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:672.out 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:673 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:673.out 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:674 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:674.out 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:675 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:675.out 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:676 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:676.out 2023-12-16T22:00:39.908 INFO:teuthology.orchestra.run.smithi028.stdout:677 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:677.out 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:678 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:678.out 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:679 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:679.out 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:680 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:680.out 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:681 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:681.out 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:682 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:682.out 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:683 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:683.out 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:684 2023-12-16T22:00:39.909 INFO:teuthology.orchestra.run.smithi028.stdout:684.out 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:685 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:685.out 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:686 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:686.out 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:687 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:687.out 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:688 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:688.out 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:689 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:689.out 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:690 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:690.out 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:691 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:691.out 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:692 2023-12-16T22:00:39.910 INFO:teuthology.orchestra.run.smithi028.stdout:692.out 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:693 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:693.out 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:694 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:694.out 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:695 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:695.out 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:696 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:696.out 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:697 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:697.out 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:698 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:698.out 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:699 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:699.out 2023-12-16T22:00:39.911 INFO:teuthology.orchestra.run.smithi028.stdout:700 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:700.out 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:701 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:701.out 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:702 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:702.out 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:703 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:703.out 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:704 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:704.out 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:705 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:705.out 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:706 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:706.out 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:707 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:707.out 2023-12-16T22:00:39.912 INFO:teuthology.orchestra.run.smithi028.stdout:708 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:708.out 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:709 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:709.out 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:710 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:710.out 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:711 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:711.out 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:712 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:712.out 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:713 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:713.out 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:714 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:714.out 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:715 2023-12-16T22:00:39.913 INFO:teuthology.orchestra.run.smithi028.stdout:715.out 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:716 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:716.out 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:717 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:717.out 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:718 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:718.out 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:719 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:719.out 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:720 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:720.out 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:721 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:721.out 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:722 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:722.out 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:723 2023-12-16T22:00:39.914 INFO:teuthology.orchestra.run.smithi028.stdout:723.out 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:724 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:724.out 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:725 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:725.out 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:726 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:726.out 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:727 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:727.out 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:728 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:728.out 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:729 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:729.out 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:730 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:730.out 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:731 2023-12-16T22:00:39.915 INFO:teuthology.orchestra.run.smithi028.stdout:731.out 2023-12-16T22:00:39.916 INFO:teuthology.orchestra.run.smithi028.stdout:732 2023-12-16T22:00:39.916 INFO:teuthology.orchestra.run.smithi028.stdout:732.out 2023-12-16T22:00:39.916 INFO:teuthology.orchestra.run.smithi028.stdout:733 2023-12-16T22:00:39.916 INFO:teuthology.orchestra.run.smithi028.stdout:733.out 2023-12-16T22:00:39.916 INFO:teuthology.orchestra.run.smithi028.stdout:group.list 2023-12-16T22:00:39.916 INFO:teuthology.orchestra.run.smithi028.stdout:Makefile 2023-12-16T22:00:39.916 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/001) 2023-12-16T22:00:40.118 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:00:40.124 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:00:40.124 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:42.556 INFO:teuthology.orchestra.run.smithi028.stdout:generic/001 2s 2023-12-16T22:00:42.565 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/001 2023-12-16T22:00:42.565 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:00:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:42.570 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi028.stderr:ceph-fuse[47505]: fuse finished with error 0 and tester_r 0 2023-12-16T22:00:42.581 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:00:42.582 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/002) 2023-12-16T22:00:42.779 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:00:42.784 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:00:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:43.228 INFO:teuthology.orchestra.run.smithi028.stdout:generic/002 1s 2023-12-16T22:00:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/002 2023-12-16T22:00:43.234 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:00:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:43.247 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:00:43.247 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/003) 2023-12-16T22:00:43.456 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:00:43.460 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:00:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:43.563 INFO:teuthology.orchestra.run.smithi028.stdout:generic/003 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:00:43.567 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/003 2023-12-16T22:00:43.567 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/003 2023-12-16T22:00:43.567 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:00:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:43.578 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:00:43.578 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/004) 2023-12-16T22:00:43.725 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:00:43.729 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:00:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:43.836 INFO:teuthology.orchestra.run.smithi028.stdout:generic/004 [not run] O_TMPFILE is not supported 2023-12-16T22:00:43.839 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/004 2023-12-16T22:00:43.840 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/004 2023-12-16T22:00:43.840 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:00:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:43.850 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:00:43.850 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/005) 2023-12-16T22:00:43.991 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:00:43.995 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:00:43.995 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:44.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:00:44.191 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:00:44.469 INFO:teuthology.orchestra.run.smithi028.stdout:generic/005 0s 2023-12-16T22:00:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/005 2023-12-16T22:00:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:00:44.476 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:00:44.485 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:00:44.485 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/006) 2023-12-16T22:00:44.667 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:00:44.671 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:00:44.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:01:14.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:01:14.316 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:01:32.746 INFO:teuthology.orchestra.run.smithi028.stdout:generic/006 48s 2023-12-16T22:01:32.751 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/006 2023-12-16T22:01:32.752 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:01:32.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:01:32.833 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:01:32.834 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/007) 2023-12-16T22:01:33.024 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:01:33.031 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:01:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:01:44.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:01:44.472 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:02:14.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:02:14.818 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:02:44.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:02:44.886 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:02:46.238 INFO:teuthology.orchestra.run.smithi028.stdout:generic/007 73s 2023-12-16T22:02:46.243 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/007 2023-12-16T22:02:46.243 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:02:46.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:02:46.254 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:02:46.254 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/008) 2023-12-16T22:02:46.440 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:02:46.444 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:02:46.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:02:47.617 INFO:teuthology.orchestra.run.smithi028.stdout:generic/008 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-12-16T22:02:47.623 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/008 2023-12-16T22:02:47.623 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/008 2023-12-16T22:02:47.623 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:02:47.627 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:02:47.640 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:02:47.640 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/009) 2023-12-16T22:02:47.833 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:02:47.838 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:02:47.838 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:02:48.982 INFO:teuthology.orchestra.run.smithi028.stdout:generic/009 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-12-16T22:02:48.987 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/009 2023-12-16T22:02:48.987 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/009 2023-12-16T22:02:48.987 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:02:48.990 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:02:49.000 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:02:49.001 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/010) 2023-12-16T22:02:49.189 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:02:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:02:49.194 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:02:49.321 INFO:teuthology.orchestra.run.smithi028.stdout:generic/010 [not run] /tmp/tmp.nC0MjoAvrDxfstests-dev/src/dbtest not built 2023-12-16T22:02:49.325 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/010 2023-12-16T22:02:49.325 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/010 2023-12-16T22:02:49.326 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:02:49.328 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:02:49.338 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:02:49.338 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/011) 2023-12-16T22:02:49.508 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:02:49.512 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:02:49.512 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:03:14.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:03:14.935 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:03:26.415 INFO:teuthology.orchestra.run.smithi028.stdout:generic/011 37s 2023-12-16T22:03:26.490 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/011 2023-12-16T22:03:26.490 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:03:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:03:26.504 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:03:26.505 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/012) 2023-12-16T22:03:26.705 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:03:26.709 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:03:26.709 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:03:27.815 INFO:teuthology.orchestra.run.smithi028.stdout:generic/012 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T22:03:27.820 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/012 2023-12-16T22:03:27.820 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/012 2023-12-16T22:03:27.820 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:03:27.823 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:03:27.832 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:03:27.833 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/013) 2023-12-16T22:03:28.043 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:03:28.047 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:03:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:03:44.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:03:44.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:04:15.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:04:15.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:04:45.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:04:45.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:05:15.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:05:15.766 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:05:16.696 INFO:teuthology.orchestra.run.smithi028.stdout:generic/013 108s 2023-12-16T22:05:16.767 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/013 2023-12-16T22:05:16.767 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:05:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:05:16.785 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:05:16.786 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/014) 2023-12-16T22:05:17.024 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:05:17.028 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:05:17.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:05:46.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:05:46.113 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:05:46.153 INFO:tasks.ceph.mds.a.smithi028.stderr:2023-12-16T22:05:46.156+0000 7fbc383f4640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 93777) UID: 0 2023-12-16T22:05:46.154 INFO:tasks.ceph.mds.c.smithi028.stderr:2023-12-16T22:05:46.156+0000 7f83ace9a640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 93777) UID: 0 2023-12-16T22:09:33.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:09:33.508 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:10:03.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:10:03.756 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:10:25.540 INFO:teuthology.orchestra.run.smithi028.stdout:generic/014 308s 2023-12-16T22:10:25.546 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/014 2023-12-16T22:10:25.546 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:25.604 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:25.605 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/015) 2023-12-16T22:10:25.892 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:25.898 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:26.070 INFO:teuthology.orchestra.run.smithi028.stdout:generic/015 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:26.075 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/015 2023-12-16T22:10:26.075 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/015 2023-12-16T22:10:26.075 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:26.077 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:26.087 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:26.088 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/016) 2023-12-16T22:10:26.343 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:26.347 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:26.347 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:27.821 INFO:teuthology.orchestra.run.smithi028.stdout:generic/016 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T22:10:27.825 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/016 2023-12-16T22:10:27.825 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/016 2023-12-16T22:10:27.825 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:27.839 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:27.839 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/017) 2023-12-16T22:10:28.427 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:28.432 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:28.578 INFO:teuthology.orchestra.run.smithi028.stdout:generic/017 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:28.582 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/017 2023-12-16T22:10:28.582 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/017 2023-12-16T22:10:28.582 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:28.598 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:28.598 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/018) 2023-12-16T22:10:28.843 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:28.849 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:29.021 INFO:teuthology.orchestra.run.smithi028.stdout:generic/018 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:29.027 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/018 2023-12-16T22:10:29.027 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/018 2023-12-16T22:10:29.027 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:29.040 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:29.040 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/019) 2023-12-16T22:10:29.345 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:29.350 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:29.509 INFO:teuthology.orchestra.run.smithi028.stdout:generic/019 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:29.519 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/019 2023-12-16T22:10:29.519 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/019 2023-12-16T22:10:29.519 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:29.522 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:29.538 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:29.538 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/020) 2023-12-16T22:10:29.776 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:29.781 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:29.781 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:33.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:10:33.879 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:10:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:generic/020 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/020.out.bad) 2023-12-16T22:10:40.887 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/020.out 2023-12-16 21:58:54.245020848 +0000 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/020.out.bad 2023-12-16 22:10:40.844767279 +0000 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -47,9 +47,13 @@ 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: user.snrub="fish2\012" 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: *** really long value 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: -0000000 00 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: -* 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: -ATTRSIZE 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: +attr_set: No space left on device 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: +Could not set "long_attr" for 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: +attr_get: No data available 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: +Could not get "long_attr" for 2023-12-16T22:10:40.888 INFO:teuthology.orchestra.run.smithi028.stdout: +0000000 2023-12-16T22:10:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: +attr_remove: No data available 2023-12-16T22:10:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: +Could not remove "long_attr" for 2023-12-16T22:10:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: *** set/get/remove really long names (expect failure) 2023-12-16T22:10:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: attr_set: Invalid argument 2023-12-16T22:10:40.889 INFO:teuthology.orchestra.run.smithi028.stdout: Could not set "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" for 2023-12-16T22:10:40.890 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/020 2023-12-16T22:10:40.890 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/020 2023-12-16T22:10:40.890 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-16T22:10:40.893 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T22:10:40.906 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-16T22:10:40.906 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-16T22:10:40.907 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/021) 2023-12-16T22:10:41.450 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:41.464 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:41.464 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:42.981 INFO:teuthology.orchestra.run.smithi028.stdout:generic/021 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T22:10:42.985 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/021 2023-12-16T22:10:42.985 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/021 2023-12-16T22:10:42.985 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:42.999 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:42.999 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/022) 2023-12-16T22:10:43.522 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:43.527 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:44.915 INFO:teuthology.orchestra.run.smithi028.stdout:generic/022 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T22:10:44.923 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/022 2023-12-16T22:10:44.923 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/022 2023-12-16T22:10:44.923 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:44.940 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:44.940 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/023) 2023-12-16T22:10:45.572 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:45.577 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:45.577 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:47.388 INFO:teuthology.orchestra.run.smithi028.stdout:generic/023 2s 2023-12-16T22:10:47.405 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/023 2023-12-16T22:10:47.405 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:47.408 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:47.512 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:47.512 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/024) 2023-12-16T22:10:47.996 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:48.001 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:48.001 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:generic/024 [not run] kernel doesn't support renameat2 syscall 2023-12-16T22:10:48.193 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/024 2023-12-16T22:10:48.193 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/024 2023-12-16T22:10:48.193 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:48.197 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:48.212 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:48.213 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/025) 2023-12-16T22:10:48.804 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:48.809 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:48.809 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:49.035 INFO:teuthology.orchestra.run.smithi028.stdout:generic/025 [not run] kernel doesn't support renameat2 syscall 2023-12-16T22:10:49.043 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/025 2023-12-16T22:10:49.043 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/025 2023-12-16T22:10:49.043 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:49.046 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:49.057 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:49.057 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/026) 2023-12-16T22:10:49.665 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:49.670 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:49.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:49.868 INFO:teuthology.orchestra.run.smithi028.stdout:generic/026 [not run] ACLs not supported by this filesystem type: ceph-fuse 2023-12-16T22:10:49.872 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/026 2023-12-16T22:10:49.872 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/026 2023-12-16T22:10:49.872 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:49.874 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:49.884 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:49.884 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/027) 2023-12-16T22:10:50.138 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:50.145 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:50.145 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:50.306 INFO:teuthology.orchestra.run.smithi028.stdout:generic/027 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:50.314 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/027 2023-12-16T22:10:50.314 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/027 2023-12-16T22:10:50.314 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:50.317 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:50.328 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:50.328 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/028) 2023-12-16T22:10:50.568 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:50.584 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:50.584 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:55.778 INFO:teuthology.orchestra.run.smithi028.stdout:generic/028 5s 2023-12-16T22:10:55.794 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/028 2023-12-16T22:10:55.794 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:55.796 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:55.846 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:55.846 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/029) 2023-12-16T22:10:56.090 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:56.095 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:56.096 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:56.245 INFO:teuthology.orchestra.run.smithi028.stdout:generic/029 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/029 2023-12-16T22:10:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/029 2023-12-16T22:10:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:56.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:56.264 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:56.264 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/030) 2023-12-16T22:10:56.468 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:56.473 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:56.473 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:56.628 INFO:teuthology.orchestra.run.smithi028.stdout:generic/030 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/030 2023-12-16T22:10:56.635 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/030 2023-12-16T22:10:56.635 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:56.651 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:56.651 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/031) 2023-12-16T22:10:56.881 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:56.886 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:56.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:57.031 INFO:teuthology.orchestra.run.smithi028.stdout:generic/031 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:57.035 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/031 2023-12-16T22:10:57.035 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/031 2023-12-16T22:10:57.035 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:57.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:57.048 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:57.048 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/032) 2023-12-16T22:10:57.278 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:57.284 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:57.284 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:57.417 INFO:teuthology.orchestra.run.smithi028.stdout:generic/032 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:57.421 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/032 2023-12-16T22:10:57.421 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/032 2023-12-16T22:10:57.421 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:57.424 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:57.434 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:57.434 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/033) 2023-12-16T22:10:57.680 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:57.684 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:57.684 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:57.820 INFO:teuthology.orchestra.run.smithi028.stdout:generic/033 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:57.825 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/033 2023-12-16T22:10:57.825 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/033 2023-12-16T22:10:57.825 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:57.828 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:57.839 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:57.839 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/034) 2023-12-16T22:10:58.083 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:58.088 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:58.088 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:58.253 INFO:teuthology.orchestra.run.smithi028.stdout:generic/034 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:10:58.258 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/034 2023-12-16T22:10:58.258 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/034 2023-12-16T22:10:58.258 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:58.265 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:58.288 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:58.288 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/035) 2023-12-16T22:10:58.520 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:58.524 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:58.524 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:58.793 INFO:teuthology.orchestra.run.smithi028.stdout:generic/035 0s 2023-12-16T22:10:58.832 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/035 2023-12-16T22:10:58.832 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:10:58.839 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:10:58.853 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:10:58.853 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/036) 2023-12-16T22:10:59.440 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:10:59.445 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:10:59.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:03.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:11:03.985 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:11:09.838 INFO:teuthology.orchestra.run.smithi028.stdout:generic/036 10s 2023-12-16T22:11:09.844 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/036 2023-12-16T22:11:09.844 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:10.057 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:10.057 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/037) 2023-12-16T22:11:10.453 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:10.457 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:10.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:10.612 INFO:teuthology.orchestra.run.smithi028.stdout:generic/037 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:10.616 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/037 2023-12-16T22:11:10.616 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/037 2023-12-16T22:11:10.616 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:10.618 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:10.632 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:10.632 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/038) 2023-12-16T22:11:10.850 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:10.855 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:10.855 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:10.984 INFO:teuthology.orchestra.run.smithi028.stdout:generic/038 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:10.994 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/038 2023-12-16T22:11:10.994 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/038 2023-12-16T22:11:10.994 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:10.997 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:11.014 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:11.014 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/039) 2023-12-16T22:11:11.255 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:11.261 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:11.261 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:11.413 INFO:teuthology.orchestra.run.smithi028.stdout:generic/039 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:11.417 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/039 2023-12-16T22:11:11.417 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/039 2023-12-16T22:11:11.417 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:11.420 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:11.440 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:11.440 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/040) 2023-12-16T22:11:11.675 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:11.684 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:11.684 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:11.847 INFO:teuthology.orchestra.run.smithi028.stdout:generic/040 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:11.851 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/040 2023-12-16T22:11:11.851 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/040 2023-12-16T22:11:11.851 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:11.854 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:11.865 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:11.865 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/041) 2023-12-16T22:11:12.139 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:12.148 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:12.148 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:12.309 INFO:teuthology.orchestra.run.smithi028.stdout:generic/041 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:12.317 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/041 2023-12-16T22:11:12.317 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/041 2023-12-16T22:11:12.317 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:12.324 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:12.341 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:12.341 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/042) 2023-12-16T22:11:12.592 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:12.601 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:12.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:12.747 INFO:teuthology.orchestra.run.smithi028.stdout:generic/042 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:12.750 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/042 2023-12-16T22:11:12.750 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/042 2023-12-16T22:11:12.751 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:12.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:12.766 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:12.766 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/043) 2023-12-16T22:11:13.033 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:13.037 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:13.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:13.194 INFO:teuthology.orchestra.run.smithi028.stdout:generic/043 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:13.198 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/043 2023-12-16T22:11:13.198 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/043 2023-12-16T22:11:13.198 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:13.201 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:13.215 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:13.215 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/044) 2023-12-16T22:11:13.441 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:13.447 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:13.447 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:13.590 INFO:teuthology.orchestra.run.smithi028.stdout:generic/044 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:13.595 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/044 2023-12-16T22:11:13.595 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/044 2023-12-16T22:11:13.595 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:13.598 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:13.608 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:13.608 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/045) 2023-12-16T22:11:13.850 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:13.854 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:13.854 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:14.003 INFO:teuthology.orchestra.run.smithi028.stdout:generic/045 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:14.010 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/045 2023-12-16T22:11:14.011 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/045 2023-12-16T22:11:14.011 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:14.023 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:14.023 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/046) 2023-12-16T22:11:14.270 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:14.276 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:14.276 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:14.431 INFO:teuthology.orchestra.run.smithi028.stdout:generic/046 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:14.435 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/046 2023-12-16T22:11:14.435 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/046 2023-12-16T22:11:14.435 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:14.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:14.447 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:14.448 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/047) 2023-12-16T22:11:14.710 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:14.716 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:14.716 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:14.870 INFO:teuthology.orchestra.run.smithi028.stdout:generic/047 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:14.874 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/047 2023-12-16T22:11:14.875 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/047 2023-12-16T22:11:14.875 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:14.888 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:14.888 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/048) 2023-12-16T22:11:15.120 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:15.125 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:15.273 INFO:teuthology.orchestra.run.smithi028.stdout:generic/048 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:15.277 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/048 2023-12-16T22:11:15.277 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/048 2023-12-16T22:11:15.277 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:15.298 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:15.299 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/049) 2023-12-16T22:11:15.504 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:15.509 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:15.657 INFO:teuthology.orchestra.run.smithi028.stdout:generic/049 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:15.662 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/049 2023-12-16T22:11:15.662 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/049 2023-12-16T22:11:15.662 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:15.678 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:15.678 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/050) 2023-12-16T22:11:15.910 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:15.915 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:16.136 INFO:teuthology.orchestra.run.smithi028.stdout:generic/050 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:16.140 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/050 2023-12-16T22:11:16.140 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/050 2023-12-16T22:11:16.140 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:16.157 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:16.157 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/051) 2023-12-16T22:11:16.391 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:16.399 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:16.399 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:16.544 INFO:teuthology.orchestra.run.smithi028.stdout:generic/051 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:16.551 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/051 2023-12-16T22:11:16.551 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/051 2023-12-16T22:11:16.551 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:16.553 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:16.573 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:16.574 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/052) 2023-12-16T22:11:16.802 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:16.810 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:16.969 INFO:teuthology.orchestra.run.smithi028.stdout:generic/052 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:16.973 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/052 2023-12-16T22:11:16.973 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/052 2023-12-16T22:11:16.973 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:16.975 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:16.991 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:16.991 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/053) 2023-12-16T22:11:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:17.242 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:17.242 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:17.403 INFO:teuthology.orchestra.run.smithi028.stdout:generic/053 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:17.407 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/053 2023-12-16T22:11:17.407 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/053 2023-12-16T22:11:17.407 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:17.410 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:17.420 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:17.420 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/054) 2023-12-16T22:11:17.659 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:17.664 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:17.664 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:17.814 INFO:teuthology.orchestra.run.smithi028.stdout:generic/054 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/054 2023-12-16T22:11:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/054 2023-12-16T22:11:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:17.822 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:17.844 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:17.844 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/055) 2023-12-16T22:11:18.111 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:18.115 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:18.115 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:18.272 INFO:teuthology.orchestra.run.smithi028.stdout:generic/055 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:18.278 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/055 2023-12-16T22:11:18.278 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/055 2023-12-16T22:11:18.279 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:18.281 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:18.291 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:18.292 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/056) 2023-12-16T22:11:18.542 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:18.551 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:18.551 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:18.711 INFO:teuthology.orchestra.run.smithi028.stdout:generic/056 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:18.715 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/056 2023-12-16T22:11:18.715 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/056 2023-12-16T22:11:18.715 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:18.718 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:18.727 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:18.728 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/057) 2023-12-16T22:11:18.979 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:18.987 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:18.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:19.157 INFO:teuthology.orchestra.run.smithi028.stdout:generic/057 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:19.161 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/057 2023-12-16T22:11:19.162 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/057 2023-12-16T22:11:19.162 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:19.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:19.173 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:19.173 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/058) 2023-12-16T22:11:19.421 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:19.426 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:19.426 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:20.913 INFO:teuthology.orchestra.run.smithi028.stdout:generic/058 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T22:11:20.918 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/058 2023-12-16T22:11:20.918 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/058 2023-12-16T22:11:20.918 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:20.931 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:20.932 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/059) 2023-12-16T22:11:21.480 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:21.485 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:21.485 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:21.640 INFO:teuthology.orchestra.run.smithi028.stdout:generic/059 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:21.644 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/059 2023-12-16T22:11:21.644 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/059 2023-12-16T22:11:21.644 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:21.647 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:21.656 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:21.656 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/060) 2023-12-16T22:11:21.890 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:21.894 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:21.894 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:23.393 INFO:teuthology.orchestra.run.smithi028.stdout:generic/060 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T22:11:23.397 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/060 2023-12-16T22:11:23.398 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/060 2023-12-16T22:11:23.398 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:23.401 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:23.413 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:23.413 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/061) 2023-12-16T22:11:23.972 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:23.976 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:25.182 INFO:teuthology.orchestra.run.smithi028.stdout:generic/061 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T22:11:25.187 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/061 2023-12-16T22:11:25.187 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/061 2023-12-16T22:11:25.187 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:25.202 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:25.202 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/062) 2023-12-16T22:11:25.722 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:25.726 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:25.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:25.846 INFO:teuthology.orchestra.run.smithi028.stdout:generic/062 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:25.850 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/062 2023-12-16T22:11:25.850 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/062 2023-12-16T22:11:25.851 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:25.863 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:25.863 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/063) 2023-12-16T22:11:26.029 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:26.034 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:27.607 INFO:teuthology.orchestra.run.smithi028.stdout:generic/063 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T22:11:27.612 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/063 2023-12-16T22:11:27.612 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/063 2023-12-16T22:11:27.612 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:27.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:27.790 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:27.790 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/064) 2023-12-16T22:11:28.122 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:28.126 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:28.225 INFO:teuthology.orchestra.run.smithi028.stdout:generic/064 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:28.230 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/064 2023-12-16T22:11:28.231 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/064 2023-12-16T22:11:28.231 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:28.241 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:28.241 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/065) 2023-12-16T22:11:28.382 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:28.386 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:28.506 INFO:teuthology.orchestra.run.smithi028.stdout:generic/065 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:28.509 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/065 2023-12-16T22:11:28.509 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/065 2023-12-16T22:11:28.509 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:28.520 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:28.520 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/066) 2023-12-16T22:11:28.665 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:28.669 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:28.798 INFO:teuthology.orchestra.run.smithi028.stdout:generic/066 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:28.801 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/066 2023-12-16T22:11:28.802 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/066 2023-12-16T22:11:28.802 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:28.812 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:28.813 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/067) 2023-12-16T22:11:28.956 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:28.960 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:29.087 INFO:teuthology.orchestra.run.smithi028.stdout:generic/067 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:29.091 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/067 2023-12-16T22:11:29.091 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/067 2023-12-16T22:11:29.091 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:29.101 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:29.101 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/068) 2023-12-16T22:11:29.241 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:29.246 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:29.246 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:29.358 INFO:teuthology.orchestra.run.smithi028.stdout:generic/068 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:29.362 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/068 2023-12-16T22:11:29.362 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/068 2023-12-16T22:11:29.362 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:29.372 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:29.373 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/069) 2023-12-16T22:11:29.514 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:29.518 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:29.518 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:29.619 INFO:teuthology.orchestra.run.smithi028.stdout:generic/069 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:11:29.622 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/069 2023-12-16T22:11:29.622 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/069 2023-12-16T22:11:29.622 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:11:29.624 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:29.633 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:11:29.633 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/070) 2023-12-16T22:11:29.781 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:11:29.785 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:11:29.785 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:11:34.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:11:34.034 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:12:04.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:12:04.470 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:12:34.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:12:34.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:13:04.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:13:04.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:13:22.756 INFO:teuthology.orchestra.run.smithi028.stdout:generic/070 113s 2023-12-16T22:13:22.761 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/070 2023-12-16T22:13:22.761 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:13:22.769 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:13:22.985 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:13:22.986 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/071) 2023-12-16T22:13:23.376 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:13:23.380 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:13:23.380 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:13:23.528 INFO:teuthology.orchestra.run.smithi028.stdout:generic/071 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:13:23.540 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/071 2023-12-16T22:13:23.540 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/071 2023-12-16T22:13:23.540 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:13:23.542 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:13:23.557 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:13:23.558 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/072) 2023-12-16T22:13:23.790 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:13:23.807 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:13:23.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:13:25.243 INFO:teuthology.orchestra.run.smithi028.stdout:generic/072 [not run] xfs_io fcollapse failed (old kernel/wrong fs?) 2023-12-16T22:13:25.251 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/072 2023-12-16T22:13:25.251 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/072 2023-12-16T22:13:25.251 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:13:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:13:25.269 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:13:25.269 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/073) 2023-12-16T22:13:25.500 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:13:25.509 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:13:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:13:25.669 INFO:teuthology.orchestra.run.smithi028.stdout:generic/073 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:13:25.677 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/073 2023-12-16T22:13:25.678 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/073 2023-12-16T22:13:25.678 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:13:25.681 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:13:25.693 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:13:25.693 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/074) 2023-12-16T22:13:25.991 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:13:25.997 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:13:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:13:34.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:13:34.737 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:14:04.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:14:04.837 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:14:34.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:14:34.951 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:15:05.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:15:05.308 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:15:35.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:15:35.401 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:16:05.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:16:05.489 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:16:35.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:16:35.608 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:17:05.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:17:05.917 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:17:36.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:17:36.495 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:18:06.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:18:06.552 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:18:36.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:18:36.601 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:19:06.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:19:07.000 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:19:37.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:19:37.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:20:07.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:20:07.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:20:37.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:20:37.277 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:21:07.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:21:07.675 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:21:37.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:21:37.740 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:22:07.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:22:07.851 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:22:37.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:22:37.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:23:08.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:23:08.058 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:23:38.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:23:38.364 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:24:08.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:24:08.455 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:24:38.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:24:38.803 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:25:08.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:25:08.970 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:25:39.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:25:39.968 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:26:10.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:26:10.044 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:26:40.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:26:40.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:27:10.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:27:10.334 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:27:40.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:27:40.397 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:28:10.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:28:10.522 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:28:40.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:28:40.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:29:10.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:29:10.661 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:29:40.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:29:40.716 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:30:10.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:30:10.795 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:30:40.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:30:40.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:31:10.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:31:10.927 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:31:40.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:31:40.994 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:32:11.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:32:11.107 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:32:41.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:32:41.473 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:33:11.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:33:11.514 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:33:41.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:33:41.573 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:34:12.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:34:12.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:34:42.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:34:42.610 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:35:12.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:35:12.921 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:35:43.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:35:43.196 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:36:13.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:36:13.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:36:13.600 INFO:tasks.ceph.osd.1.smithi028.stderr:2023-12-16T22:36:13.599+0000 7f56e1017640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 125310) UID: 0 2023-12-16T22:36:13.601 INFO:tasks.ceph.osd.2.smithi028.stderr:2023-12-16T22:36:13.599+0000 7fbf6e1ec640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 125310) UID: 0 2023-12-16T22:36:13.601 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-16T22:36:13.599+0000 7f4d3794e640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 125310) UID: 0 2023-12-16T22:36:13.601 INFO:tasks.ceph.osd.3.smithi028.stderr:2023-12-16T22:36:13.599+0000 7fa1baf1d640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 125310) UID: 0 2023-12-16T22:40:17.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:40:17.323 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:40:17.452 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-16T22:40:17.447+0000 7f702cf56640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45096) UID: 0 2023-12-16T22:40:17.453 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-16T22:40:17.447+0000 7f0e01b73640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45096) UID: 0 2023-12-16T22:40:17.453 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-16T22:40:17.447+0000 7f5af7bed640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45096) UID: 0 2023-12-16T22:40:17.457 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-16T22:40:17.455+0000 7f66974e8640 -1 Fail to open '/proc/45096/cmdline' error = (2) No such file or directory 2023-12-16T22:40:17.457 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-16T22:40:17.455+0000 7f66974e8640 -1 received signal: Hangup from (PID: 45096) UID: 0 2023-12-16T22:43:21.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:43:21.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:43:51.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:43:51.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:43:51.538 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-16T22:43:51.535+0000 7f66974e8640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45126) UID: 0 2023-12-16T22:43:51.538 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-16T22:43:51.535+0000 7f0e01b73640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45126) UID: 0 2023-12-16T22:43:51.539 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-16T22:43:51.535+0000 7f702cf56640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45126) UID: 0 2023-12-16T22:43:51.539 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-16T22:43:51.535+0000 7f5af7bed640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45126) UID: 0 2023-12-16T22:44:54.436 INFO:teuthology.orchestra.run.smithi028.stdout:generic/074 1888s 2023-12-16T22:44:54.516 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/074 2023-12-16T22:44:54.516 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:44:54.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:44:54.535 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:44:54.535 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/075) 2023-12-16T22:44:55.469 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:44:55.474 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:44:55.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:19.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:46:19.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:46:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:generic/075 85s 2023-12-16T22:46:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/075 2023-12-16T22:46:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:20.038 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:20.288 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:20.289 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/076) 2023-12-16T22:46:20.611 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:20.615 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:20.615 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:20.719 INFO:teuthology.orchestra.run.smithi028.stdout:generic/076 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:46:20.722 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/076 2023-12-16T22:46:20.722 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/076 2023-12-16T22:46:20.722 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:20.733 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:20.733 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/077) 2023-12-16T22:46:20.884 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:20.888 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:20.888 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:22.007 INFO:teuthology.orchestra.run.smithi028.stdout:generic/077 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:46:22.013 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/077 2023-12-16T22:46:22.013 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/077 2023-12-16T22:46:22.013 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:22.017 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:22.030 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:22.030 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/078) 2023-12-16T22:46:22.227 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:22.232 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:22.232 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:22.399 INFO:teuthology.orchestra.run.smithi028.stdout:generic/078 [not run] kernel doesn't support renameat2 syscall 2023-12-16T22:46:22.403 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/078 2023-12-16T22:46:22.403 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/078 2023-12-16T22:46:22.404 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:22.406 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:22.415 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:22.416 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/079) 2023-12-16T22:46:22.943 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:22.947 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:22.947 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:23.118 INFO:teuthology.orchestra.run.smithi028.stdout:generic/079 [not run] file system doesn't support chattr +ia 2023-12-16T22:46:23.121 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/079 2023-12-16T22:46:23.122 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/079 2023-12-16T22:46:23.122 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:23.125 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:23.132 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:23.132 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/080) 2023-12-16T22:46:23.281 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:23.287 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:23.287 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:25.475 INFO:teuthology.orchestra.run.smithi028.stdout:generic/080 2s 2023-12-16T22:46:25.483 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/080 2023-12-16T22:46:25.484 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:25.500 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:25.501 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/081) 2023-12-16T22:46:25.740 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:25.745 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:25.873 INFO:teuthology.orchestra.run.smithi028.stdout:generic/081 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:46:25.878 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/081 2023-12-16T22:46:25.878 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/081 2023-12-16T22:46:25.878 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:25.890 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:25.891 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/082) 2023-12-16T22:46:26.073 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:26.077 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:26.077 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:26.206 INFO:teuthology.orchestra.run.smithi028.stdout:generic/082 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:46:26.211 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/082 2023-12-16T22:46:26.211 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/082 2023-12-16T22:46:26.211 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:26.223 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:26.224 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/083) 2023-12-16T22:46:26.395 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:26.400 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:26.523 INFO:teuthology.orchestra.run.smithi028.stdout:generic/083 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:46:26.527 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/083 2023-12-16T22:46:26.528 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/083 2023-12-16T22:46:26.528 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:26.540 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:26.540 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/084) 2023-12-16T22:46:26.698 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:26.702 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:26.702 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:26.804 INFO:teuthology.orchestra.run.smithi028.stdout:generic/084 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:46:26.807 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/084 2023-12-16T22:46:26.807 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/084 2023-12-16T22:46:26.808 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:26.810 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:26.818 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:26.818 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/085) 2023-12-16T22:46:26.960 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:26.964 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:26.964 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:27.096 INFO:teuthology.orchestra.run.smithi028.stdout:generic/085 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T22:46:27.099 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/085 2023-12-16T22:46:27.099 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/085 2023-12-16T22:46:27.099 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:27.102 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:27.110 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:27.110 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/086) 2023-12-16T22:46:27.253 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:27.257 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:27.257 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:32.246 INFO:teuthology.orchestra.run.smithi028.stdout:generic/086 5s 2023-12-16T22:46:32.409 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/086 2023-12-16T22:46:32.409 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:32.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:32.482 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:32.482 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/087) 2023-12-16T22:46:34.060 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:34.064 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:34.405 INFO:teuthology.orchestra.run.smithi028.stdout:generic/087 0s 2023-12-16T22:46:34.410 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/087 2023-12-16T22:46:34.410 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:34.412 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:34.419 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:34.420 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/088) 2023-12-16T22:46:34.927 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:34.931 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:34.931 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:35.093 INFO:teuthology.orchestra.run.smithi028.stdout:generic/088 1s 2023-12-16T22:46:35.097 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/088 2023-12-16T22:46:35.097 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T22:46:35.099 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:35.108 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T22:46:35.108 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/089) 2023-12-16T22:46:35.614 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T22:46:35.617 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T22:46:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T22:46:49.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:46:49.963 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:47:21.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:47:21.042 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:47:51.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:47:51.112 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:48:21.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:48:21.260 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:48:51.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:48:51.541 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:49:21.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:49:21.705 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:49:52.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:49:52.020 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:50:22.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:50:22.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:50:22.090 INFO:tasks.ceph.mds.a.smithi028.stderr:2023-12-16T22:50:22.084+0000 7fbc383f4640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 133517) UID: 0 2023-12-16T22:50:22.091 INFO:tasks.ceph.mds.c.smithi028.stderr:2023-12-16T22:50:22.084+0000 7f83ace9a640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 133517) UID: 0 2023-12-16T22:53:22.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:53:22.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:53:52.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:53:52.517 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:54:22.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:54:22.826 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:54:52.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:54:52.910 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:55:23.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:55:23.025 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:55:23.037 INFO:tasks.ceph.mds.a.smithi028.stderr:2023-12-16T22:55:23.036+0000 7fbc383f4640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 133754) UID: 0 2023-12-16T22:55:23.037 INFO:tasks.ceph.mds.c.smithi028.stderr:2023-12-16T22:55:23.036+0000 7f83ace9a640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 133754) UID: 0 2023-12-16T22:58:15.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:58:15.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:58:46.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:58:46.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:59:16.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:59:16.358 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:59:46.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T22:59:46.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:00:16.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:00:16.461 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:00:16.483 INFO:tasks.ceph.mds.a.smithi028.stderr:2023-12-16T23:00:16.480+0000 7fbc383f4640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 135786) UID: 0 2023-12-16T23:00:16.483 INFO:tasks.ceph.mds.c.smithi028.stderr:2023-12-16T23:00:16.480+0000 7f83ace9a640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 135786) UID: 0 2023-12-16T23:02:59.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:02:59.652 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:02:59.676 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-16T23:02:59.671+0000 7f702cf56640 -1 Fail to open '/proc/45207/cmdline' error = (2) No such file or directory 2023-12-16T23:02:59.676 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-16T23:02:59.671+0000 7f702cf56640 -1 received signal: Hangup from (PID: 45207) UID: 0 2023-12-16T23:02:59.677 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-16T23:02:59.671+0000 7f5af7bed640 -1 Fail to open '/proc/45207/cmdline' error = (2) No such file or directory 2023-12-16T23:02:59.677 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-16T23:02:59.671+0000 7f5af7bed640 -1 received signal: Hangup from (PID: 45207) UID: 0 2023-12-16T23:02:59.677 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-16T23:02:59.671+0000 7f66974e8640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45207) UID: 0 2023-12-16T23:02:59.679 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-16T23:02:59.671+0000 7f0e01b73640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 45207) UID: 0 2023-12-16T23:04:23.284 INFO:teuthology.orchestra.run.smithi028.stdout:generic/089 1068s 2023-12-16T23:04:23.343 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/089 2023-12-16T23:04:23.343 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:04:23.347 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:04:23.362 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:04:23.363 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/090) 2023-12-16T23:04:24.349 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:04:24.355 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:04:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:04:24.599 INFO:teuthology.orchestra.run.smithi028.stdout:generic/090 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:04:24.606 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/090 2023-12-16T23:04:24.606 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/090 2023-12-16T23:04:24.606 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:04:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:04:24.620 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:04:24.621 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/091) 2023-12-16T23:04:24.818 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:04:24.822 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:04:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:05:23.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:05:23.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:05:23.946 INFO:tasks.ceph.osd.1.smithi028.stderr:2023-12-16T23:05:23.944+0000 7f56e1017640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 136662) UID: 0 2023-12-16T23:05:23.946 INFO:tasks.ceph.osd.2.smithi028.stderr:2023-12-16T23:05:23.944+0000 7fbf6e1ec640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 136662) UID: 0 2023-12-16T23:05:23.947 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-16T23:05:23.944+0000 7f4d3794e640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 136662) UID: 0 2023-12-16T23:05:23.947 INFO:tasks.ceph.osd.3.smithi028.stderr:2023-12-16T23:05:23.944+0000 7fa1baf1d640 -1 received signal: Hangup from killall ceph-osd -1 (PID: 136662) UID: 0 2023-12-16T23:08:03.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:08:03.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:08:33.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:08:33.108 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:08:50.336 INFO:teuthology.orchestra.run.smithi028.stdout:generic/091 266s 2023-12-16T23:08:50.343 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/091 2023-12-16T23:08:50.343 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:50.348 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:50.360 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:50.361 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/092) 2023-12-16T23:08:50.975 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:50.980 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:50.980 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:51.186 INFO:teuthology.orchestra.run.smithi028.stdout:generic/092 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-16T23:08:51.190 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/092 2023-12-16T23:08:51.190 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/092 2023-12-16T23:08:51.190 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:51.193 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:51.203 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:51.203 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/093) 2023-12-16T23:08:51.727 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:51.731 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:51.731 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:52.645 INFO:teuthology.orchestra.run.smithi028.stdout:generic/093 1s 2023-12-16T23:08:52.651 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/093 2023-12-16T23:08:52.651 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:52.653 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:52.662 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:52.663 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/094) 2023-12-16T23:08:53.160 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:53.164 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:53.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:53.266 INFO:teuthology.orchestra.run.smithi028.stdout:generic/094 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:08:53.269 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/094 2023-12-16T23:08:53.269 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/094 2023-12-16T23:08:53.270 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:53.272 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:53.280 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:53.280 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/095) 2023-12-16T23:08:53.426 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:53.430 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:53.431 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:53.531 INFO:teuthology.orchestra.run.smithi028.stdout:generic/095 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:08:53.535 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/095 2023-12-16T23:08:53.535 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/095 2023-12-16T23:08:53.535 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:53.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:53.545 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:53.546 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/096) 2023-12-16T23:08:53.693 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:53.697 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:53.697 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:53.796 INFO:teuthology.orchestra.run.smithi028.stdout:generic/096 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:08:53.800 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/096 2023-12-16T23:08:53.800 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/096 2023-12-16T23:08:53.800 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:53.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:53.811 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:53.812 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/097) 2023-12-16T23:08:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:53.959 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:54.618 INFO:teuthology.orchestra.run.smithi028.stdout:generic/097 1s 2023-12-16T23:08:54.623 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/097 2023-12-16T23:08:54.623 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:54.633 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:54.633 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/098) 2023-12-16T23:08:54.778 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:54.782 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:54.782 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:54.886 INFO:teuthology.orchestra.run.smithi028.stdout:generic/098 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:08:54.890 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/098 2023-12-16T23:08:54.890 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/098 2023-12-16T23:08:54.890 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:54.893 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:54.901 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:54.901 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/099) 2023-12-16T23:08:55.046 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:55.049 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:55.050 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:55.192 INFO:teuthology.orchestra.run.smithi028.stdout:generic/099 [not run] ACLs not supported by this filesystem type: ceph-fuse 2023-12-16T23:08:55.196 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/099 2023-12-16T23:08:55.196 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/099 2023-12-16T23:08:55.196 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:08:55.198 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:08:55.206 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:08:55.206 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/100) 2023-12-16T23:08:55.352 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:08:55.357 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:08:55.357 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:03.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:09:03.148 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:09:33.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:09:33.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:09:37.447 INFO:teuthology.orchestra.run.smithi028.stdout:generic/100 42s 2023-12-16T23:09:37.455 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/100 2023-12-16T23:09:37.456 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:37.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:37.479 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:37.480 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/101) 2023-12-16T23:09:37.675 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:37.681 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:37.681 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:38.223 INFO:teuthology.orchestra.run.smithi028.stdout:generic/101 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:38.227 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/101 2023-12-16T23:09:38.227 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/101 2023-12-16T23:09:38.227 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:38.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:38.238 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:38.238 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/102) 2023-12-16T23:09:38.387 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:38.391 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:38.493 INFO:teuthology.orchestra.run.smithi028.stdout:generic/102 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:38.496 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/102 2023-12-16T23:09:38.496 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/102 2023-12-16T23:09:38.496 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:38.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:38.507 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:38.507 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/103) 2023-12-16T23:09:38.656 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:38.660 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:38.660 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:38.761 INFO:teuthology.orchestra.run.smithi028.stdout:generic/103 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:38.764 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/103 2023-12-16T23:09:38.764 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/103 2023-12-16T23:09:38.765 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:38.767 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:38.775 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:38.775 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/104) 2023-12-16T23:09:38.931 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:38.934 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:39.051 INFO:teuthology.orchestra.run.smithi028.stdout:generic/104 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:39.055 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/104 2023-12-16T23:09:39.055 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/104 2023-12-16T23:09:39.055 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:39.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:39.065 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:39.066 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/105) 2023-12-16T23:09:39.214 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:39.218 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:39.218 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:39.321 INFO:teuthology.orchestra.run.smithi028.stdout:generic/105 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:39.324 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/105 2023-12-16T23:09:39.324 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/105 2023-12-16T23:09:39.325 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:39.327 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:39.334 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:39.335 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/106) 2023-12-16T23:09:39.488 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:39.492 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:39.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:39.607 INFO:teuthology.orchestra.run.smithi028.stdout:generic/106 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:39.611 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/106 2023-12-16T23:09:39.611 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/106 2023-12-16T23:09:39.611 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:39.613 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:39.621 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:39.622 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/107) 2023-12-16T23:09:39.771 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:39.774 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:39.775 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:39.887 INFO:teuthology.orchestra.run.smithi028.stdout:generic/107 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/107 2023-12-16T23:09:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/107 2023-12-16T23:09:39.891 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:39.893 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:39.902 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:39.902 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/108) 2023-12-16T23:09:40.055 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:40.059 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:40.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:40.350 INFO:teuthology.orchestra.run.smithi028.stdout:generic/108 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:40.356 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/108 2023-12-16T23:09:40.356 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/108 2023-12-16T23:09:40.356 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:40.360 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:40.372 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:40.372 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/109) 2023-12-16T23:09:40.578 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:40.582 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:40.694 INFO:teuthology.orchestra.run.smithi028.stdout:generic/109 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:09:40.698 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/109 2023-12-16T23:09:40.698 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/109 2023-12-16T23:09:40.698 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:40.700 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:40.709 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:40.709 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/110) 2023-12-16T23:09:40.862 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:40.866 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:40.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:41.018 INFO:teuthology.orchestra.run.smithi028.stdout:generic/110 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:09:41.022 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/110 2023-12-16T23:09:41.022 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/110 2023-12-16T23:09:41.022 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:41.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:41.032 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:41.032 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/111) 2023-12-16T23:09:41.553 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:41.557 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:41.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:41.705 INFO:teuthology.orchestra.run.smithi028.stdout:generic/111 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:09:41.709 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/111 2023-12-16T23:09:41.709 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/111 2023-12-16T23:09:41.709 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:09:41.711 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:09:41.719 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:09:41.719 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/112) 2023-12-16T23:09:42.228 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:09:42.233 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:09:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:10:03.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:10:03.296 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:10:33.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:10:33.463 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:11:03.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:11:03.535 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:11:13.793 INFO:teuthology.orchestra.run.smithi028.stdout:generic/112 91s 2023-12-16T23:11:13.801 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/112 2023-12-16T23:11:13.802 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:13.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:13.991 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:13.991 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/113) 2023-12-16T23:11:14.389 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:14.393 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:14.393 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:33.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:11:33.583 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:11:52.762 INFO:teuthology.orchestra.run.smithi028.stdout:generic/113 38s 2023-12-16T23:11:52.767 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/113 2023-12-16T23:11:52.767 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:52.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:52.969 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:52.969 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/114) 2023-12-16T23:11:53.314 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:53.318 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:53.318 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:53.646 INFO:teuthology.orchestra.run.smithi028.stdout:generic/114 [not run] device block size: 4096 greater than 512 2023-12-16T23:11:53.652 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/114 2023-12-16T23:11:53.653 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/114 2023-12-16T23:11:53.653 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:53.655 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:53.666 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:53.667 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/115) 2023-12-16T23:11:53.842 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:53.846 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:53.998 INFO:teuthology.orchestra.run.smithi028.stdout:generic/115 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:11:54.002 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/115 2023-12-16T23:11:54.002 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/115 2023-12-16T23:11:54.002 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:54.004 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:54.012 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:54.012 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/116) 2023-12-16T23:11:54.524 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:54.527 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:54.528 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:54.679 INFO:teuthology.orchestra.run.smithi028.stdout:generic/116 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:11:54.682 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/116 2023-12-16T23:11:54.682 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/116 2023-12-16T23:11:54.682 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:54.692 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:54.692 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/117) 2023-12-16T23:11:55.204 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:55.208 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:55.309 INFO:teuthology.orchestra.run.smithi028.stdout:generic/117 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:11:55.313 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/117 2023-12-16T23:11:55.313 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/117 2023-12-16T23:11:55.313 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:55.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:55.324 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:55.324 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/118) 2023-12-16T23:11:55.471 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:55.475 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:55.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:55.626 INFO:teuthology.orchestra.run.smithi028.stdout:generic/118 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:11:55.629 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/118 2023-12-16T23:11:55.630 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/118 2023-12-16T23:11:55.630 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:55.632 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:55.640 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:55.640 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/119) 2023-12-16T23:11:56.122 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:56.126 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:56.126 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:56.276 INFO:teuthology.orchestra.run.smithi028.stdout:generic/119 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:11:56.280 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/119 2023-12-16T23:11:56.280 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/119 2023-12-16T23:11:56.280 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:56.282 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:56.290 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:56.291 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/120) 2023-12-16T23:11:56.805 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:56.808 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:56.808 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:56.907 INFO:teuthology.orchestra.run.smithi028.stdout:generic/120 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:11:56.911 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/120 2023-12-16T23:11:56.911 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/120 2023-12-16T23:11:56.911 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:56.913 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:56.921 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:56.921 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/121) 2023-12-16T23:11:57.066 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:57.070 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:57.070 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:57.223 INFO:teuthology.orchestra.run.smithi028.stdout:generic/121 [not run] Dedupe not supported by test filesystem type: ceph-fuse 2023-12-16T23:11:57.226 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/121 2023-12-16T23:11:57.227 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/121 2023-12-16T23:11:57.227 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:57.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:57.237 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:57.237 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/122) 2023-12-16T23:11:57.756 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:57.760 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:57.760 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:57.912 INFO:teuthology.orchestra.run.smithi028.stdout:generic/122 [not run] Dedupe not supported by test filesystem type: ceph-fuse 2023-12-16T23:11:57.915 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/122 2023-12-16T23:11:57.915 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/122 2023-12-16T23:11:57.915 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:57.917 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:57.925 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:57.925 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/123) 2023-12-16T23:11:58.451 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:58.455 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:58.455 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:58.797 INFO:teuthology.orchestra.run.smithi028.stdout:generic/123 0s 2023-12-16T23:11:58.802 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/123 2023-12-16T23:11:58.802 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:58.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:58.812 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:58.813 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/124) 2023-12-16T23:11:59.329 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:59.332 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:59.333 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:59.438 INFO:teuthology.orchestra.run.smithi028.stdout:generic/124 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:11:59.442 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/124 2023-12-16T23:11:59.442 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/124 2023-12-16T23:11:59.442 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:11:59.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:11:59.452 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:11:59.452 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/125) 2023-12-16T23:11:59.600 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:11:59.605 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:11:59.605 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:12:03.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:12:03.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:12:33.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:12:33.948 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:13:00.089 INFO:teuthology.orchestra.run.smithi028.stdout:generic/125 61s 2023-12-16T23:13:00.097 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/125 2023-12-16T23:13:00.097 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:13:00.101 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:13:00.129 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:13:00.130 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/126) 2023-12-16T23:13:00.652 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:13:00.656 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:13:00.656 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:13:01.219 INFO:teuthology.orchestra.run.smithi028.stdout:generic/126 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/126.out.bad) 2023-12-16T23:13:01.221 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/126.out 2023-12-16 21:58:54.253020513 +0000 2023-12-16T23:13:01.221 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/126.out.bad 2023-12-16 23:13:01.215412130 +0000 2023-12-16T23:13:01.221 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,7 +1,7 @@ 2023-12-16T23:13:01.221 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 126 2023-12-16T23:13:01.221 INFO:teuthology.orchestra.run.smithi028.stdout: -x a 001 file owned by (99/99) as user/group(12/100) PASS 2023-12-16T23:13:01.221 INFO:teuthology.orchestra.run.smithi028.stdout: -x a 010 file owned by (99/99) as user/group(200/99) PASS 2023-12-16T23:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout: -x a 100 file owned by (99/99) as user/group(99/500) PASS 2023-12-16T23:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout: +x a 001 file owned by (99/99) as user/group(12/100) FAIL 2023-12-16T23:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout: +x a 010 file owned by (99/99) as user/group(200/99) FAIL 2023-12-16T23:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout: +x a 100 file owned by (99/99) as user/group(99/500) FAIL 2023-12-16T23:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout: w a 002 file owned by (99/99) as user/group(12/100) PASS 2023-12-16T23:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout: w a 020 file owned by (99/99) as user/group(200/99) PASS 2023-12-16T23:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout: w a 200 file owned by (99/99) as user/group(99/500) PASS 2023-12-16T23:13:01.223 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/126 2023-12-16T23:13:01.223 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/126 2023-12-16T23:13:01.224 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-16T23:13:01.226 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:13:01.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T23:13:01.233 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-16T23:13:01.234 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-16T23:13:01.234 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/127) 2023-12-16T23:13:01.775 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:13:01.779 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:13:01.779 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:13:04.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:13:04.073 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:13:34.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:13:34.205 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:14:04.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:14:04.330 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:14:34.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:14:34.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:15:04.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:15:04.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:15:34.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:15:34.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:16:04.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:16:04.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:16:34.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:16:34.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:17:04.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:17:04.899 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:17:35.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:17:35.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:18:05.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:18:05.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:18:35.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:18:35.146 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:19:05.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:19:05.271 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:19:35.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:19:35.418 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:20:05.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:20:05.542 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:20:35.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:20:35.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:21:05.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:21:05.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:21:35.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:21:35.699 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:22:05.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:22:05.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:22:35.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:22:35.898 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:23:05.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:23:05.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:23:36.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:23:36.136 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:24:06.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:24:06.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:24:36.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:24:36.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:25:06.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:25:06.349 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:25:36.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:25:36.441 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:26:06.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:26:06.489 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:26:36.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:26:36.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:27:06.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:27:06.586 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:27:36.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:27:36.635 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:28:06.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:28:06.792 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:28:36.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:28:36.907 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:29:07.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:29:07.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:29:37.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:29:37.114 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:30:07.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:30:07.186 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:30:37.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:30:37.585 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:31:07.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:31:07.635 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:31:37.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:31:37.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:32:07.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:32:07.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:32:07.830 INFO:tasks.ceph.mds.a.smithi028.stderr:2023-12-16T23:32:07.827+0000 7fbc383f4640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 166566) UID: 0 2023-12-16T23:32:07.830 INFO:tasks.ceph.mds.c.smithi028.stderr:2023-12-16T23:32:07.827+0000 7f83ace9a640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 166566) UID: 0 2023-12-16T23:34:43.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:34:43.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:35:13.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:35:13.291 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:35:43.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:35:43.366 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:36:13.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:36:13.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:36:43.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:36:43.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:37:13.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:37:13.915 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:37:43.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:37:43.963 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:38:14.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:38:14.328 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:38:44.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:38:44.394 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:39:14.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:39:14.512 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:39:44.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:39:44.586 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:40:14.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:40:14.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:40:44.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:40:44.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:41:14.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:41:14.789 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:41:44.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:41:44.856 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:42:14.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:42:14.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:42:45.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:42:45.112 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:43:15.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:43:15.177 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:43:44.342 INFO:teuthology.orchestra.run.smithi028.stdout:generic/127 1843s 2023-12-16T23:43:44.346 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/127 2023-12-16T23:43:44.347 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:43:44.350 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:44.363 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:43:44.363 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/128) 2023-12-16T23:43:44.954 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:43:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:43:44.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:45.099 INFO:teuthology.orchestra.run.smithi028.stdout:generic/128 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:43:45.104 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/128 2023-12-16T23:43:45.104 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/128 2023-12-16T23:43:45.105 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:43:45.107 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:45.118 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:43:45.118 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/129) 2023-12-16T23:43:45.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:43:45.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:43:45.301 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:43:45.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:45.431 INFO:teuthology.orchestra.run.smithi028.stdout:generic/129 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:43:45.435 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/129 2023-12-16T23:43:45.435 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/129 2023-12-16T23:43:45.435 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:43:45.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:45.449 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:43:45.449 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/130) 2023-12-16T23:43:45.622 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:43:45.626 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:43:45.626 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:45.732 INFO:teuthology.orchestra.run.smithi028.stdout:generic/130 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:43:45.735 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/130 2023-12-16T23:43:45.735 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/130 2023-12-16T23:43:45.735 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:43:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:45.746 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:43:45.747 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/131) 2023-12-16T23:43:45.895 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:43:45.899 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:43:45.899 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:47.340 INFO:teuthology.orchestra.run.smithi028.stdout:generic/131 2s 2023-12-16T23:43:47.347 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/131 2023-12-16T23:43:47.347 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:43:47.350 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:47.361 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:43:47.361 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/132) 2023-12-16T23:43:47.913 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:43:47.917 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:43:47.917 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:48.018 INFO:teuthology.orchestra.run.smithi028.stdout:generic/132 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:43:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/132 2023-12-16T23:43:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/132 2023-12-16T23:43:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:43:48.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:43:48.032 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:43:48.032 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/133) 2023-12-16T23:43:48.179 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:43:48.183 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:43:48.183 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:44:15.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:44:15.265 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:44:45.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:44:45.348 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:45:15.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:45:15.464 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:45:45.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:45:45.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:46:15.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:46:15.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:46:45.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:46:45.637 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:47:15.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:47:15.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:47:45.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:47:45.733 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:48:15.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:48:15.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:48:45.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:48:45.998 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:49:16.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:49:16.288 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:49:46.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:49:46.336 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:50:16.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:50:16.436 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:50:46.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:50:46.492 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:51:16.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:51:16.667 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:51:46.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:51:46.824 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:52:17.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:52:17.040 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:52:47.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:52:47.092 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:52:51.634 INFO:teuthology.orchestra.run.smithi028.stdout:generic/133 543s 2023-12-16T23:52:51.642 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/133 2023-12-16T23:52:51.642 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:51.646 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:51.661 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:51.661 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/134) 2023-12-16T23:52:52.225 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:52.229 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:52.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:52.393 INFO:teuthology.orchestra.run.smithi028.stdout:generic/134 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:52.397 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/134 2023-12-16T23:52:52.397 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/134 2023-12-16T23:52:52.397 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:52.399 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:52.407 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:52.408 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/135) 2023-12-16T23:52:53.046 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:53.053 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:53.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:53.216 INFO:teuthology.orchestra.run.smithi028.stdout:generic/135 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:52:53.220 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/135 2023-12-16T23:52:53.220 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/135 2023-12-16T23:52:53.220 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:53.222 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:53.231 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:53.231 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/136) 2023-12-16T23:52:53.379 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:53.383 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:53.384 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:53.580 INFO:teuthology.orchestra.run.smithi028.stdout:generic/136 [not run] Dedupe not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:53.584 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/136 2023-12-16T23:52:53.584 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/136 2023-12-16T23:52:53.584 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:53.587 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:53.595 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:53.595 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/137) 2023-12-16T23:52:54.101 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:54.105 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:54.252 INFO:teuthology.orchestra.run.smithi028.stdout:generic/137 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:54.256 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/137 2023-12-16T23:52:54.256 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/137 2023-12-16T23:52:54.256 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:54.266 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:54.266 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/138) 2023-12-16T23:52:54.794 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:54.946 INFO:teuthology.orchestra.run.smithi028.stdout:generic/138 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:54.950 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/138 2023-12-16T23:52:54.950 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/138 2023-12-16T23:52:54.950 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:54.953 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:54.961 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:54.961 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/139) 2023-12-16T23:52:55.486 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:55.490 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:55.633 INFO:teuthology.orchestra.run.smithi028.stdout:generic/139 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:55.636 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/139 2023-12-16T23:52:55.637 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/139 2023-12-16T23:52:55.637 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:55.639 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:55.647 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:55.647 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/140) 2023-12-16T23:52:56.175 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:56.179 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:56.330 INFO:teuthology.orchestra.run.smithi028.stdout:generic/140 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:56.333 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/140 2023-12-16T23:52:56.333 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/140 2023-12-16T23:52:56.333 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:56.335 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:56.344 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:56.344 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/141) 2023-12-16T23:52:56.884 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:56.888 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:56.888 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:56.990 INFO:teuthology.orchestra.run.smithi028.stdout:generic/141 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:52:56.994 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/141 2023-12-16T23:52:56.994 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/141 2023-12-16T23:52:56.994 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:56.996 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:57.004 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:57.005 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/142) 2023-12-16T23:52:57.150 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:57.154 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:57.154 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:57.301 INFO:teuthology.orchestra.run.smithi028.stdout:generic/142 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:57.304 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/142 2023-12-16T23:52:57.304 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/142 2023-12-16T23:52:57.305 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:57.307 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:57.315 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:57.315 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/143) 2023-12-16T23:52:57.817 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:57.820 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:57.821 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:57.971 INFO:teuthology.orchestra.run.smithi028.stdout:generic/143 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/143 2023-12-16T23:52:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/143 2023-12-16T23:52:57.975 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:57.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:57.986 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:57.986 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/144) 2023-12-16T23:52:58.473 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:58.477 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:58.620 INFO:teuthology.orchestra.run.smithi028.stdout:generic/144 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:58.624 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/144 2023-12-16T23:52:58.624 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/144 2023-12-16T23:52:58.624 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:58.626 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:58.634 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:58.634 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/145) 2023-12-16T23:52:59.158 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:59.162 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:59.162 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:59.307 INFO:teuthology.orchestra.run.smithi028.stdout:generic/145 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:59.311 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/145 2023-12-16T23:52:59.311 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/145 2023-12-16T23:52:59.311 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:59.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:59.322 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:59.322 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/146) 2023-12-16T23:52:59.833 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:52:59.837 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:52:59.837 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:59.984 INFO:teuthology.orchestra.run.smithi028.stdout:generic/146 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:52:59.988 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/146 2023-12-16T23:52:59.988 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/146 2023-12-16T23:52:59.988 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:52:59.990 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:52:59.998 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:52:59.998 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/147) 2023-12-16T23:53:00.509 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:00.513 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:00.513 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:00.656 INFO:teuthology.orchestra.run.smithi028.stdout:generic/147 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:00.660 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/147 2023-12-16T23:53:00.660 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/147 2023-12-16T23:53:00.660 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:00.662 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:00.670 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:00.671 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/148) 2023-12-16T23:53:01.186 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:01.190 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:01.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:01.335 INFO:teuthology.orchestra.run.smithi028.stdout:generic/148 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/148 2023-12-16T23:53:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/148 2023-12-16T23:53:01.339 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:01.341 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:01.350 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:01.350 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/149) 2023-12-16T23:53:01.858 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:01.862 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:01.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:02.002 INFO:teuthology.orchestra.run.smithi028.stdout:generic/149 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:02.006 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/149 2023-12-16T23:53:02.006 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/149 2023-12-16T23:53:02.006 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:02.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:02.016 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:02.017 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/150) 2023-12-16T23:53:02.520 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:02.524 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:02.524 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:02.672 INFO:teuthology.orchestra.run.smithi028.stdout:generic/150 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:02.676 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/150 2023-12-16T23:53:02.676 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/150 2023-12-16T23:53:02.676 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:02.678 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:02.686 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:02.687 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/151) 2023-12-16T23:53:03.214 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:03.218 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:03.218 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:03.363 INFO:teuthology.orchestra.run.smithi028.stdout:generic/151 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:03.366 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/151 2023-12-16T23:53:03.366 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/151 2023-12-16T23:53:03.367 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:03.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:03.377 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:03.377 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/152) 2023-12-16T23:53:03.902 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:03.906 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:03.906 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:04.051 INFO:teuthology.orchestra.run.smithi028.stdout:generic/152 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:04.054 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/152 2023-12-16T23:53:04.055 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/152 2023-12-16T23:53:04.055 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:04.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:04.065 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:04.065 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/153) 2023-12-16T23:53:04.552 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:04.557 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:04.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:04.707 INFO:teuthology.orchestra.run.smithi028.stdout:generic/153 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:04.711 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/153 2023-12-16T23:53:04.711 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/153 2023-12-16T23:53:04.711 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:04.713 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:04.722 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:04.722 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/154) 2023-12-16T23:53:05.230 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:05.234 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:05.379 INFO:teuthology.orchestra.run.smithi028.stdout:generic/154 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:05.382 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/154 2023-12-16T23:53:05.382 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/154 2023-12-16T23:53:05.382 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:05.393 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:05.393 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/155) 2023-12-16T23:53:05.908 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:05.912 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:06.059 INFO:teuthology.orchestra.run.smithi028.stdout:generic/155 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/155 2023-12-16T23:53:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/155 2023-12-16T23:53:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:06.074 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:06.074 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/156) 2023-12-16T23:53:06.586 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:06.754 INFO:teuthology.orchestra.run.smithi028.stdout:generic/156 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:06.758 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/156 2023-12-16T23:53:06.758 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/156 2023-12-16T23:53:06.758 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:06.769 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:06.769 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/157) 2023-12-16T23:53:07.290 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:07.294 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:07.294 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:07.440 INFO:teuthology.orchestra.run.smithi028.stdout:generic/157 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:07.444 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/157 2023-12-16T23:53:07.444 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/157 2023-12-16T23:53:07.444 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:07.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:07.454 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:07.455 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/158) 2023-12-16T23:53:07.973 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:07.977 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:07.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:08.139 INFO:teuthology.orchestra.run.smithi028.stdout:generic/158 [not run] Dedupe not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:08.142 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/158 2023-12-16T23:53:08.142 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/158 2023-12-16T23:53:08.142 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:08.153 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:08.153 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/159) 2023-12-16T23:53:08.664 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:08.668 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:08.668 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:08.822 INFO:teuthology.orchestra.run.smithi028.stdout:generic/159 [not run] file system doesn't support chattr +i 2023-12-16T23:53:08.826 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/159 2023-12-16T23:53:08.826 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/159 2023-12-16T23:53:08.826 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:08.829 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:08.837 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:08.837 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/160) 2023-12-16T23:53:08.983 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:08.988 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:08.988 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:generic/160 [not run] file system doesn't support chattr +i 2023-12-16T23:53:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/160 2023-12-16T23:53:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/160 2023-12-16T23:53:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:09.128 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:09.136 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:09.136 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/161) 2023-12-16T23:53:09.280 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:09.284 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:09.285 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:09.385 INFO:teuthology.orchestra.run.smithi028.stdout:generic/161 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:09.389 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/161 2023-12-16T23:53:09.389 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/161 2023-12-16T23:53:09.389 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:09.391 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:09.399 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:09.415 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/162) 2023-12-16T23:53:09.559 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:09.563 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:09.563 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:09.669 INFO:teuthology.orchestra.run.smithi028.stdout:generic/162 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:09.672 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/162 2023-12-16T23:53:09.673 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/162 2023-12-16T23:53:09.673 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:09.675 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:09.683 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:09.683 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/163) 2023-12-16T23:53:09.829 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:09.833 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:09.833 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:09.935 INFO:teuthology.orchestra.run.smithi028.stdout:generic/163 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:09.938 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/163 2023-12-16T23:53:09.939 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/163 2023-12-16T23:53:09.939 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:09.941 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:09.949 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:09.949 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/164) 2023-12-16T23:53:10.093 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:10.097 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:10.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:10.205 INFO:teuthology.orchestra.run.smithi028.stdout:generic/164 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:10.208 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/164 2023-12-16T23:53:10.209 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/164 2023-12-16T23:53:10.209 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:10.219 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:10.219 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/165) 2023-12-16T23:53:10.366 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:10.370 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:10.370 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:10.474 INFO:teuthology.orchestra.run.smithi028.stdout:generic/165 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:10.478 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/165 2023-12-16T23:53:10.478 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/165 2023-12-16T23:53:10.478 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:10.480 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:10.488 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:10.489 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/166) 2023-12-16T23:53:10.632 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:10.636 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:10.636 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:10.737 INFO:teuthology.orchestra.run.smithi028.stdout:generic/166 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:10.741 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/166 2023-12-16T23:53:10.741 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/166 2023-12-16T23:53:10.741 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:10.743 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:10.751 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:10.751 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/167) 2023-12-16T23:53:10.897 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:10.901 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:11.002 INFO:teuthology.orchestra.run.smithi028.stdout:generic/167 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:11.006 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/167 2023-12-16T23:53:11.006 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/167 2023-12-16T23:53:11.006 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:11.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:11.017 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:11.017 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/168) 2023-12-16T23:53:11.160 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:11.165 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:11.165 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:11.266 INFO:teuthology.orchestra.run.smithi028.stdout:generic/168 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:11.270 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/168 2023-12-16T23:53:11.270 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/168 2023-12-16T23:53:11.270 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:11.272 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:11.280 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:11.280 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/169) 2023-12-16T23:53:11.424 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:11.428 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:11.428 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:11.528 INFO:teuthology.orchestra.run.smithi028.stdout:generic/169 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:11.531 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/169 2023-12-16T23:53:11.531 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/169 2023-12-16T23:53:11.531 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:11.533 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:11.541 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:11.542 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/170) 2023-12-16T23:53:11.685 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:11.689 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:11.689 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:11.791 INFO:teuthology.orchestra.run.smithi028.stdout:generic/170 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:11.796 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/170 2023-12-16T23:53:11.796 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/170 2023-12-16T23:53:11.796 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:11.798 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:11.806 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:11.807 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/171) 2023-12-16T23:53:11.951 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:11.955 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:12.079 INFO:teuthology.orchestra.run.smithi028.stdout:generic/171 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:12.082 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/171 2023-12-16T23:53:12.082 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/171 2023-12-16T23:53:12.082 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:12.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:12.092 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:12.093 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/172) 2023-12-16T23:53:12.239 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:12.243 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:12.352 INFO:teuthology.orchestra.run.smithi028.stdout:generic/172 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:12.355 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/172 2023-12-16T23:53:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/172 2023-12-16T23:53:12.356 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:12.358 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:12.366 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:12.366 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/173) 2023-12-16T23:53:12.514 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:12.518 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:12.519 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:12.624 INFO:teuthology.orchestra.run.smithi028.stdout:generic/173 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:12.628 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/173 2023-12-16T23:53:12.628 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/173 2023-12-16T23:53:12.628 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:12.630 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:12.639 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:12.639 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/174) 2023-12-16T23:53:12.784 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:12.787 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:12.787 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:12.896 INFO:teuthology.orchestra.run.smithi028.stdout:generic/174 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:12.900 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/174 2023-12-16T23:53:12.900 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/174 2023-12-16T23:53:12.900 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:12.902 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:12.910 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:12.910 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/175) 2023-12-16T23:53:13.054 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:13.058 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:13.058 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:13.160 INFO:teuthology.orchestra.run.smithi028.stdout:generic/175 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:13.164 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/175 2023-12-16T23:53:13.165 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/175 2023-12-16T23:53:13.165 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:13.166 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:13.174 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:13.174 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/176) 2023-12-16T23:53:13.317 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:13.321 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:13.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:13.423 INFO:teuthology.orchestra.run.smithi028.stdout:generic/176 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:13.426 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/176 2023-12-16T23:53:13.426 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/176 2023-12-16T23:53:13.426 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:13.429 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:13.437 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:13.437 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/177) 2023-12-16T23:53:13.580 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:13.583 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:13.584 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:13.743 INFO:teuthology.orchestra.run.smithi028.stdout:generic/177 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:13.747 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/177 2023-12-16T23:53:13.747 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/177 2023-12-16T23:53:13.747 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:13.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:13.758 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:13.758 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/178) 2023-12-16T23:53:13.904 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:13.908 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:13.908 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:14.058 INFO:teuthology.orchestra.run.smithi028.stdout:generic/178 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:14.062 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/178 2023-12-16T23:53:14.062 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/178 2023-12-16T23:53:14.062 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:14.064 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:14.073 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:14.073 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/179) 2023-12-16T23:53:14.588 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:14.592 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:14.592 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:14.740 INFO:teuthology.orchestra.run.smithi028.stdout:generic/179 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:14.744 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/179 2023-12-16T23:53:14.744 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/179 2023-12-16T23:53:14.744 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:14.746 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:14.754 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:14.755 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/180) 2023-12-16T23:53:15.265 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:15.269 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:15.419 INFO:teuthology.orchestra.run.smithi028.stdout:generic/180 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:15.424 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/180 2023-12-16T23:53:15.424 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/180 2023-12-16T23:53:15.424 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:15.433 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:15.433 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/181) 2023-12-16T23:53:15.920 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:15.923 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:16.067 INFO:teuthology.orchestra.run.smithi028.stdout:generic/181 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:16.071 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/181 2023-12-16T23:53:16.071 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/181 2023-12-16T23:53:16.071 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:16.081 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:16.081 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/182) 2023-12-16T23:53:16.597 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:16.601 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:16.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:16.754 INFO:teuthology.orchestra.run.smithi028.stdout:generic/182 [not run] Dedupe not supported by test filesystem type: ceph-fuse 2023-12-16T23:53:16.757 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/182 2023-12-16T23:53:16.757 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/182 2023-12-16T23:53:16.757 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:16.768 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:16.768 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/183) 2023-12-16T23:53:17.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:53:17.227 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:53:17.275 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:17.279 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:17.279 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:17.383 INFO:teuthology.orchestra.run.smithi028.stdout:generic/183 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:17.386 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/183 2023-12-16T23:53:17.386 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/183 2023-12-16T23:53:17.386 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:17.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:17.397 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:17.397 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/184) 2023-12-16T23:53:17.583 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:17.586 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:17.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:17.738 INFO:teuthology.orchestra.run.smithi028.stdout:generic/184 [failed, exit status 1]- output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/184.out.bad) 2023-12-16T23:53:17.740 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/184.out 2023-12-16 21:58:54.257020346 +0000 2023-12-16T23:53:17.740 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/184.out.bad 2023-12-16 23:53:17.739849263 +0000 2023-12-16T23:53:17.740 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1 +1,2 @@ 2023-12-16T23:53:17.741 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 184 2023-12-16T23:53:17.741 INFO:teuthology.orchestra.run.smithi028.stdout: +/tmp/tmp.nC0MjoAvrDxfstests-dev/tests/generic/184: line 23: /tmp/tmp.XR9uc7m4Witest/null: Permission denied 2023-12-16T23:53:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/184 2023-12-16T23:53:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/184 2023-12-16T23:53:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-16T23:53:17.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T23:53:17.753 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-16T23:53:17.753 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-16T23:53:17.753 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/185) 2023-12-16T23:53:17.899 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:17.903 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:17.903 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:18.003 INFO:teuthology.orchestra.run.smithi028.stdout:generic/185 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:18.007 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/185 2023-12-16T23:53:18.007 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/185 2023-12-16T23:53:18.007 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:18.017 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:18.017 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/186) 2023-12-16T23:53:18.162 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:18.166 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:18.167 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:18.267 INFO:teuthology.orchestra.run.smithi028.stdout:generic/186 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:18.271 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/186 2023-12-16T23:53:18.271 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/186 2023-12-16T23:53:18.271 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:18.273 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:18.281 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:18.281 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/187) 2023-12-16T23:53:18.424 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:18.428 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:18.530 INFO:teuthology.orchestra.run.smithi028.stdout:generic/187 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:18.533 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/187 2023-12-16T23:53:18.533 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/187 2023-12-16T23:53:18.533 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:18.535 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:18.543 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:18.544 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/188) 2023-12-16T23:53:18.688 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:18.692 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:18.795 INFO:teuthology.orchestra.run.smithi028.stdout:generic/188 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:18.798 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/188 2023-12-16T23:53:18.799 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/188 2023-12-16T23:53:18.799 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:18.801 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:18.809 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:18.809 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/189) 2023-12-16T23:53:18.953 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:18.957 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:18.957 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:19.058 INFO:teuthology.orchestra.run.smithi028.stdout:generic/189 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:19.062 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/189 2023-12-16T23:53:19.062 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/189 2023-12-16T23:53:19.062 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:19.064 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:19.073 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:19.073 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/190) 2023-12-16T23:53:19.217 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:19.221 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:19.221 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:19.330 INFO:teuthology.orchestra.run.smithi028.stdout:generic/190 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:19.333 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/190 2023-12-16T23:53:19.333 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/190 2023-12-16T23:53:19.333 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:19.344 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:19.344 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/191) 2023-12-16T23:53:19.485 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:19.489 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:19.489 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:19.699 INFO:teuthology.orchestra.run.smithi028.stdout:generic/191 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:19.703 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/191 2023-12-16T23:53:19.704 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/191 2023-12-16T23:53:19.704 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:19.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:19.717 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:19.718 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/192) 2023-12-16T23:53:19.888 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:19.892 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:19.892 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:generic/192 [not run] atime not maintained by ceph-fuse 2023-12-16T23:53:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/192 2023-12-16T23:53:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/192 2023-12-16T23:53:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:20.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:20.017 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:20.018 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/193) 2023-12-16T23:53:20.163 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:20.167 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:20.167 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:20.863 INFO:teuthology.orchestra.run.smithi028.stdout:generic/193 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/193.out.bad) 2023-12-16T23:53:20.865 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/193.out 2023-12-16 21:58:54.257020346 +0000 2023-12-16T23:53:20.865 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/193.out.bad 2023-12-16 23:53:20.843765940 +0000 2023-12-16T23:53:20.865 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -24,7 +24,6 @@ 2023-12-16T23:53:20.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:20.866 INFO:teuthology.orchestra.run.smithi028.stdout: user: chmod a+r on qa_user owned file (should succeed) 2023-12-16T23:53:20.866 INFO:teuthology.orchestra.run.smithi028.stdout: user: chmod a+r on root owned file (should fail) 2023-12-16T23:53:20.866 INFO:teuthology.orchestra.run.smithi028.stdout: -chmod: changing permissions of 'test.root': Operation not permitted 2023-12-16T23:53:20.866 INFO:teuthology.orchestra.run.smithi028.stdout: check that the sgid bit is cleared 2023-12-16T23:53:20.866 INFO:teuthology.orchestra.run.smithi028.stdout: -rw-rw-rw- 2023-12-16T23:53:20.866 INFO:teuthology.orchestra.run.smithi028.stdout: check that suid bit is not cleared 2023-12-16T23:53:20.868 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/193 2023-12-16T23:53:20.868 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/193 2023-12-16T23:53:20.868 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-16T23:53:20.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T23:53:20.878 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-16T23:53:20.878 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-16T23:53:20.878 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/194) 2023-12-16T23:53:21.384 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:21.388 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:21.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:21.503 INFO:teuthology.orchestra.run.smithi028.stdout:generic/194 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:21.507 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/194 2023-12-16T23:53:21.507 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/194 2023-12-16T23:53:21.507 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:21.509 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:21.518 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:21.518 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/195) 2023-12-16T23:53:21.663 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:21.667 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:21.667 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:21.774 INFO:teuthology.orchestra.run.smithi028.stdout:generic/195 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:21.778 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/195 2023-12-16T23:53:21.778 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/195 2023-12-16T23:53:21.778 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:21.780 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:21.789 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:21.789 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/196) 2023-12-16T23:53:21.934 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:21.937 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:21.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:22.039 INFO:teuthology.orchestra.run.smithi028.stdout:generic/196 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:22.043 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/196 2023-12-16T23:53:22.043 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/196 2023-12-16T23:53:22.043 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:22.045 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:22.054 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:22.054 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/197) 2023-12-16T23:53:22.200 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:22.204 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:22.204 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:22.307 INFO:teuthology.orchestra.run.smithi028.stdout:generic/197 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:22.311 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/197 2023-12-16T23:53:22.311 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/197 2023-12-16T23:53:22.311 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:22.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:22.321 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:22.321 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/198) 2023-12-16T23:53:22.467 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:22.471 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:22.471 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:23.512 INFO:teuthology.orchestra.run.smithi028.stdout:generic/198 1s 2023-12-16T23:53:23.520 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/198 2023-12-16T23:53:23.520 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:23.523 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:23.670 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:23.670 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/199) 2023-12-16T23:53:24.142 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:24.146 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:24.146 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:24.247 INFO:teuthology.orchestra.run.smithi028.stdout:generic/199 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:24.251 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/199 2023-12-16T23:53:24.251 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/199 2023-12-16T23:53:24.251 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:24.262 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:24.262 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/200) 2023-12-16T23:53:24.408 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:24.412 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:24.515 INFO:teuthology.orchestra.run.smithi028.stdout:generic/200 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:24.519 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/200 2023-12-16T23:53:24.519 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/200 2023-12-16T23:53:24.519 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:24.529 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:24.529 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/201) 2023-12-16T23:53:24.676 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:24.680 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:24.680 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:24.783 INFO:teuthology.orchestra.run.smithi028.stdout:generic/201 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:24.787 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/201 2023-12-16T23:53:24.787 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/201 2023-12-16T23:53:24.787 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:24.797 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:24.797 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/202) 2023-12-16T23:53:24.942 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:24.946 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:25.060 INFO:teuthology.orchestra.run.smithi028.stdout:generic/202 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:25.064 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/202 2023-12-16T23:53:25.064 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/202 2023-12-16T23:53:25.064 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:25.075 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:25.075 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/203) 2023-12-16T23:53:25.226 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:25.230 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:25.230 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:25.346 INFO:teuthology.orchestra.run.smithi028.stdout:generic/203 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:25.349 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/203 2023-12-16T23:53:25.350 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/203 2023-12-16T23:53:25.350 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:25.352 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:25.360 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:25.360 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/204) 2023-12-16T23:53:25.578 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:25.582 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:25.582 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:26.756 INFO:teuthology.orchestra.run.smithi028.stdout:generic/204 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:26.762 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/204 2023-12-16T23:53:26.762 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/204 2023-12-16T23:53:26.762 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:26.767 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:26.779 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:26.780 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/205) 2023-12-16T23:53:26.972 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:26.976 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:26.976 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:27.088 INFO:teuthology.orchestra.run.smithi028.stdout:generic/205 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:27.092 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/205 2023-12-16T23:53:27.092 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/205 2023-12-16T23:53:27.092 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:27.094 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:27.103 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:27.103 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/206) 2023-12-16T23:53:27.258 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:27.262 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:27.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:27.371 INFO:teuthology.orchestra.run.smithi028.stdout:generic/206 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:53:27.375 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/206 2023-12-16T23:53:27.375 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/206 2023-12-16T23:53:27.375 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:53:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:27.387 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:53:27.387 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/207) 2023-12-16T23:53:27.534 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:53:27.538 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:53:27.538 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:53:47.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:53:47.344 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:54:10.022 INFO:teuthology.orchestra.run.smithi028.stdout:generic/207 43s 2023-12-16T23:54:10.029 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/207 2023-12-16T23:54:10.030 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:54:10.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:54:10.047 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:54:10.047 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/208) 2023-12-16T23:54:10.582 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:54:10.586 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:54:10.587 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:54:17.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:54:17.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:54:47.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:54:47.463 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:55:17.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:55:17.740 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:55:47.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:55:47.831 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:56:17.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:56:17.891 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:56:48.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:56:48.171 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:57:18.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:57:18.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:57:30.968 INFO:teuthology.orchestra.run.smithi028.stdout:generic/208 200s 2023-12-16T23:57:30.974 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/208 2023-12-16T23:57:30.974 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:57:30.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:30.988 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:57:30.988 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/209) 2023-12-16T23:57:31.550 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:57:31.554 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:57:31.555 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:48.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:57:48.300 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:57:51.928 INFO:teuthology.orchestra.run.smithi028.stdout:generic/209 [failed, exit status 1]- output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/209.out.bad) 2023-12-16T23:57:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/209.out 2023-12-16 21:58:54.261020178 +0000 2023-12-16T23:57:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/209.out.bad 2023-12-16 23:57:51.900521882 +0000 2023-12-16T23:57:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,2 +1,2 @@ 2023-12-16T23:57:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 209 2023-12-16T23:57:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: -test ran for 30 seconds without error 2023-12-16T23:57:51.936 INFO:teuthology.orchestra.run.smithi028.stdout: +reader found old byte at pos 4096 2023-12-16T23:57:51.937 INFO:teuthology.orchestra.run.smithi028.stdout: \ No newline at end of file 2023-12-16T23:57:51.940 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/209 2023-12-16T23:57:51.941 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/209 2023-12-16T23:57:51.941 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-16T23:57:51.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T23:57:51.993 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-16T23:57:51.993 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-16T23:57:51.993 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/210) 2023-12-16T23:57:52.453 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:57:52.457 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:57:52.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:52.787 INFO:teuthology.orchestra.run.smithi028.stdout:generic/210 0s 2023-12-16T23:57:52.794 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/210 2023-12-16T23:57:52.794 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:57:52.798 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:52.830 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:57:52.830 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/211) 2023-12-16T23:57:53.332 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:57:53.336 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:57:53.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:53.699 INFO:teuthology.orchestra.run.smithi028.stdout:generic/211 0s 2023-12-16T23:57:53.705 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/211 2023-12-16T23:57:53.706 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:57:53.708 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:53.719 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:57:53.719 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/212) 2023-12-16T23:57:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:57:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:57:53.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:54.127 INFO:teuthology.orchestra.run.smithi028.stdout:generic/212 1s 2023-12-16T23:57:54.133 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/212 2023-12-16T23:57:54.134 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:57:54.136 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:54.145 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:57:54.145 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/213) 2023-12-16T23:57:54.296 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:57:54.300 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:57:54.300 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:54.698 INFO:teuthology.orchestra.run.smithi028.stdout:generic/213 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/213.out.bad) 2023-12-16T23:57:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/213.out 2023-12-16 21:58:54.261020178 +0000 2023-12-16T23:57:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/213.out.bad 2023-12-16 23:57:54.700447358 +0000 2023-12-16T23:57:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,4 +1,3 @@ 2023-12-16T23:57:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 213 2023-12-16T23:57:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: We should get: fallocate: No space left on device 2023-12-16T23:57:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: Strangely, xfs_io sometimes says "Success" when something went wrong, FYI 2023-12-16T23:57:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: -fallocate: No space left on device 2023-12-16T23:57:54.703 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/213 2023-12-16T23:57:54.703 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/213 2023-12-16T23:57:54.703 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-16T23:57:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T23:57:54.732 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-16T23:57:54.732 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-16T23:57:54.732 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/214) 2023-12-16T23:57:55.315 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:57:55.319 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:57:55.319 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:55.897 INFO:teuthology.orchestra.run.smithi028.stdout:generic/214 0s 2023-12-16T23:57:55.903 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/214 2023-12-16T23:57:55.903 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:57:55.905 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:55.915 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:57:55.915 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/215) 2023-12-16T23:57:56.545 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:57:56.549 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:57:56.549 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:59.766 INFO:teuthology.orchestra.run.smithi028.stdout:generic/215 3s 2023-12-16T23:57:59.772 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/215 2023-12-16T23:57:59.772 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:57:59.775 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:57:59.793 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:57:59.796 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/216) 2023-12-16T23:58:00.053 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:00.058 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:00.058 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:00.211 INFO:teuthology.orchestra.run.smithi028.stdout:generic/216 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:00.215 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/216 2023-12-16T23:58:00.215 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/216 2023-12-16T23:58:00.215 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:00.222 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:00.236 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:00.236 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/217) 2023-12-16T23:58:00.536 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:00.540 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:00.541 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:00.690 INFO:teuthology.orchestra.run.smithi028.stdout:generic/217 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/217 2023-12-16T23:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/217 2023-12-16T23:58:00.695 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:00.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:00.720 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:00.720 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/218) 2023-12-16T23:58:00.982 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:00.988 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:01.151 INFO:teuthology.orchestra.run.smithi028.stdout:generic/218 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/218 2023-12-16T23:58:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/218 2023-12-16T23:58:01.155 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:01.158 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:01.178 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:01.178 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/219) 2023-12-16T23:58:01.443 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:01.454 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:01.454 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:01.604 INFO:teuthology.orchestra.run.smithi028.stdout:generic/219 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:01.608 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/219 2023-12-16T23:58:01.608 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/219 2023-12-16T23:58:01.608 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:01.611 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:01.631 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:01.632 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/220) 2023-12-16T23:58:01.933 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:01.938 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:01.939 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:02.118 INFO:teuthology.orchestra.run.smithi028.stdout:generic/220 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:02.124 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/220 2023-12-16T23:58:02.124 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/220 2023-12-16T23:58:02.124 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:02.145 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:02.145 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/221) 2023-12-16T23:58:02.374 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:02.378 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:02.378 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:03.669 INFO:teuthology.orchestra.run.smithi028.stdout:generic/221 1s 2023-12-16T23:58:03.675 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/221 2023-12-16T23:58:03.675 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:03.677 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:03.686 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:03.686 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/222) 2023-12-16T23:58:03.938 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:03.944 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:03.944 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:04.079 INFO:teuthology.orchestra.run.smithi028.stdout:generic/222 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:04.083 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/222 2023-12-16T23:58:04.083 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/222 2023-12-16T23:58:04.084 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:04.086 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:04.097 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:04.097 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/223) 2023-12-16T23:58:04.336 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:04.340 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:04.340 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:04.493 INFO:teuthology.orchestra.run.smithi028.stdout:generic/223 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:04.498 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/223 2023-12-16T23:58:04.498 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/223 2023-12-16T23:58:04.498 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:04.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:04.520 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:04.520 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/224) 2023-12-16T23:58:04.772 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:04.781 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:04.781 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:04.969 INFO:teuthology.orchestra.run.smithi028.stdout:generic/224 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:04.973 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/224 2023-12-16T23:58:04.974 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/224 2023-12-16T23:58:04.974 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:04.992 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:04.992 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/225) 2023-12-16T23:58:05.241 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:05.246 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:05.414 INFO:teuthology.orchestra.run.smithi028.stdout:generic/225 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:05.428 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/225 2023-12-16T23:58:05.428 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/225 2023-12-16T23:58:05.428 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:05.446 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:05.446 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/226) 2023-12-16T23:58:05.701 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:05.705 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:05.854 INFO:teuthology.orchestra.run.smithi028.stdout:generic/226 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:05.864 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/226 2023-12-16T23:58:05.864 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/226 2023-12-16T23:58:05.864 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:05.878 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:05.878 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/227) 2023-12-16T23:58:06.144 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:06.149 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:06.316 INFO:teuthology.orchestra.run.smithi028.stdout:generic/227 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:06.320 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/227 2023-12-16T23:58:06.320 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/227 2023-12-16T23:58:06.321 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:06.336 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:06.336 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/228) 2023-12-16T23:58:06.575 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:06.584 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:06.841 INFO:teuthology.orchestra.run.smithi028.stdout:generic/228 0s 2023-12-16T23:58:06.846 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/228 2023-12-16T23:58:06.846 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:06.863 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:06.863 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/229) 2023-12-16T23:58:07.467 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:07.476 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:07.476 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:07.636 INFO:teuthology.orchestra.run.smithi028.stdout:generic/229 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:07.640 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/229 2023-12-16T23:58:07.640 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/229 2023-12-16T23:58:07.641 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:07.660 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:07.661 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/230) 2023-12-16T23:58:07.913 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:07.918 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:07.918 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:08.096 INFO:teuthology.orchestra.run.smithi028.stdout:generic/230 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:08.101 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/230 2023-12-16T23:58:08.101 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/230 2023-12-16T23:58:08.101 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:08.103 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:08.113 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:08.114 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/231) 2023-12-16T23:58:08.398 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:08.403 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:08.403 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:08.574 INFO:teuthology.orchestra.run.smithi028.stdout:generic/231 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:08.578 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/231 2023-12-16T23:58:08.578 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/231 2023-12-16T23:58:08.578 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:08.581 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:08.604 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:08.604 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/232) 2023-12-16T23:58:08.866 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:08.873 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:08.873 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:09.033 INFO:teuthology.orchestra.run.smithi028.stdout:generic/232 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:09.037 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/232 2023-12-16T23:58:09.037 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/232 2023-12-16T23:58:09.037 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:09.040 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:09.058 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:09.058 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/233) 2023-12-16T23:58:09.295 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:09.449 INFO:teuthology.orchestra.run.smithi028.stdout:generic/233 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:09.454 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/233 2023-12-16T23:58:09.454 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/233 2023-12-16T23:58:09.454 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:09.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:09.467 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:09.467 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/234) 2023-12-16T23:58:09.722 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:09.726 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:09.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:09.874 INFO:teuthology.orchestra.run.smithi028.stdout:generic/234 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:09.878 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/234 2023-12-16T23:58:09.878 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/234 2023-12-16T23:58:09.878 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:09.881 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:09.894 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:09.895 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/235) 2023-12-16T23:58:10.144 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:10.149 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:10.149 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:10.308 INFO:teuthology.orchestra.run.smithi028.stdout:generic/235 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:10.312 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/235 2023-12-16T23:58:10.312 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/235 2023-12-16T23:58:10.312 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:10.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:10.326 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:10.326 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/236) 2023-12-16T23:58:10.580 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:10.590 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:10.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:11.909 INFO:teuthology.orchestra.run.smithi028.stdout:generic/236 1s 2023-12-16T23:58:11.920 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/236 2023-12-16T23:58:11.920 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:11.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:12.069 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:12.069 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/237) 2023-12-16T23:58:12.533 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:12.538 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:12.538 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:12.777 INFO:teuthology.orchestra.run.smithi028.stdout:generic/237 [not run] ACLs not supported by this filesystem type: ceph-fuse 2023-12-16T23:58:12.780 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/237 2023-12-16T23:58:12.780 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/237 2023-12-16T23:58:12.780 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:12.783 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:12.796 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:12.796 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/238) 2023-12-16T23:58:13.055 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:13.061 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:13.062 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:13.218 INFO:teuthology.orchestra.run.smithi028.stdout:generic/238 [not run] this test requires a valid $SCRATCH_DEV 2023-12-16T23:58:13.222 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/238 2023-12-16T23:58:13.222 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/238 2023-12-16T23:58:13.222 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-16T23:58:13.225 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:13.244 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-16T23:58:13.244 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/239) 2023-12-16T23:58:13.506 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-16T23:58:13.512 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-16T23:58:13.512 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-16T23:58:18.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:58:18.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:58:48.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:58:48.547 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:59:18.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:59:18.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:59:48.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T23:59:48.814 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-17T00:00:09.346 INFO:tasks.ceph.mon.a.smithi028.stderr:2023-12-17T00:00:09.348+0000 7fbb0e295640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 214016) UID: 0 2023-12-17T00:00:09.346 INFO:tasks.ceph.mon.a.smithi028.stderr:2023-12-17T00:00:09.348+0000 7fbb0e295640 -1 mon.a@0(leader) e1 *** Got Signal Hangup *** 2023-12-17T00:00:09.346 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-17T00:00:09.348+0000 7f4d3794e640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 214016) UID: 0 2023-12-17T00:00:09.346 INFO:tasks.ceph.osd.2.smithi028.stderr:2023-12-17T00:00:09.348+0000 7fbf6e1ec640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 214016) UID: 0 2023-12-17T00:00:09.347 INFO:tasks.ceph.osd.1.smithi028.stderr:2023-12-17T00:00:09.348+0000 7f56e1017640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 214016) UID: 0 2023-12-17T00:00:09.347 INFO:tasks.ceph.mds.a.smithi028.stderr:2023-12-17T00:00:09.348+0000 7fbc383f4640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 214016) UID: 0 2023-12-17T00:00:09.347 INFO:tasks.ceph.osd.3.smithi028.stderr:2023-12-17T00:00:09.348+0000 7fa1baf1d640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 214016) UID: 0 2023-12-17T00:00:09.348 INFO:tasks.ceph.mds.c.smithi028.stderr:2023-12-17T00:00:09.348+0000 7f83ace9a640 -1 received signal: Hangup from (PID: 214016) UID: 0 2023-12-17T00:00:09.357 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-17T00:00:09.360+0000 7ff18d304640 -1 Fail to open '/proc/214016/cmdline' error = (2) No such file or directory 2023-12-17T00:00:09.357 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-17T00:00:09.360+0000 7ff18d304640 -1 received signal: Hangup from (PID: 214016) UID: 0 2023-12-17T00:00:09.380 INFO:tasks.ceph.mgr.y.smithi028.stderr:2023-12-17T00:00:09.380+0000 7ff18d304640 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw|rbd-mirror|cephfs-mirror (PID: 214028) UID: 0 2023-12-17T00:00:09.380 INFO:tasks.ceph.osd.1.smithi028.stderr:2023-12-17T00:00:09.380+0000 7f56e1017640 -1 received signal: Hangup from (PID: 214028) UID: 0 2023-12-17T00:00:09.380 INFO:tasks.ceph.mds.a.smithi028.stderr:2023-12-17T00:00:09.380+0000 7fbc383f4640 -1 received signal: Hangup from (PID: 214028) UID: 0 2023-12-17T00:00:09.380 INFO:tasks.ceph.mds.c.smithi028.stderr:2023-12-17T00:00:09.380+0000 7f83ace9a640 -1 received signal: Hangup from (PID: 214028) UID: 0 2023-12-17T00:00:09.381 INFO:tasks.ceph.mon.a.smithi028.stderr:2023-12-17T00:00:09.380+0000 7fbb0e295640 -1 received signal: Hangup from pkill -1 -x ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw|rbd-mirror|cephfs-mirror (PID: 214028) UID: 0 2023-12-17T00:00:09.381 INFO:tasks.ceph.mon.a.smithi028.stderr:2023-12-17T00:00:09.380+0000 7fbb0e295640 -1 mon.a@0(leader) e1 *** Got Signal Hangup *** 2023-12-17T00:00:09.381 INFO:tasks.ceph.osd.3.smithi028.stderr:2023-12-17T00:00:09.380+0000 7fa1baf1d640 -1 received signal: Hangup from (PID: 214028) UID: 0 2023-12-17T00:00:09.381 INFO:tasks.ceph.osd.2.smithi028.stderr:2023-12-17T00:00:09.380+0000 7fbf6e1ec640 -1 received signal: Hangup from (PID: 214028) UID: 0 2023-12-17T00:00:09.382 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-17T00:00:09.380+0000 7f4d3794e640 -1 Fail to open '/proc/214028/cmdline' error = (2) No such file or directory 2023-12-17T00:00:09.382 INFO:tasks.ceph.osd.0.smithi028.stderr:2023-12-17T00:00:09.380+0000 7f4d3794e640 -1 received signal: Hangup from (PID: 214028) UID: 0 2023-12-17T00:00:10.580 INFO:tasks.ceph.mon.c.smithi122.stderr:2023-12-17T00:00:10.580+0000 7f0502745640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45565) UID: 0 2023-12-17T00:00:10.580 INFO:tasks.ceph.mon.c.smithi122.stderr:2023-12-17T00:00:10.580+0000 7f0502745640 -1 mon.c@2(peon) e1 *** Got Signal Hangup *** 2023-12-17T00:00:10.581 INFO:tasks.ceph.mds.b.smithi122.stderr:2023-12-17T00:00:10.580+0000 7f438bb6e640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45565) UID: 0 2023-12-17T00:00:10.581 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-17T00:00:10.584+0000 7f5af7bed640 -1 received signal: Hangup from (PID: 45565) UID: 0 2023-12-17T00:00:10.581 INFO:tasks.ceph.mds.d.smithi122.stderr:2023-12-17T00:00:10.584+0000 7f8daba10640 -1 received signal: Hangup from (PID: 45565) UID: 0 2023-12-17T00:00:10.582 INFO:tasks.ceph.mon.b.smithi122.stderr:2023-12-17T00:00:10.584+0000 7fa93e641640 -1 received signal: Hangup from (PID: 45565) UID: 0 2023-12-17T00:00:10.582 INFO:tasks.ceph.mon.b.smithi122.stderr:2023-12-17T00:00:10.584+0000 7fa93e641640 -1 mon.b@1(peon) e1 *** Got Signal Hangup *** 2023-12-17T00:00:10.582 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-17T00:00:10.584+0000 7fca5b9c9640 -1 received signal: Hangup from (PID: 45565) UID: 0 2023-12-17T00:00:10.582 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-17T00:00:10.584+0000 7f702cf56640 -1 received signal: Hangup from (PID: 45565) UID: 0 2023-12-17T00:00:10.583 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-17T00:00:10.580+0000 7f66974e8640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45565) UID: 0 2023-12-17T00:00:10.605 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-17T00:00:10.588+0000 7f0e01b73640 -1 Fail to open '/proc/45565/cmdline' error = (2) No such file or directory 2023-12-17T00:00:10.605 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-17T00:00:10.588+0000 7f0e01b73640 -1 received signal: Hangup from (PID: 45565) UID: 0 2023-12-17T00:00:10.617 INFO:tasks.ceph.mon.c.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f0502745640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:10.617 INFO:tasks.ceph.mon.c.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f0502745640 -1 mon.c@2(peon) e1 *** Got Signal Hangup *** 2023-12-17T00:00:10.617 INFO:tasks.ceph.mgr.x.smithi122.stderr:2023-12-17T00:00:10.620+0000 7fca5b9c9640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:10.618 INFO:tasks.ceph.mds.b.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f438bb6e640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:10.618 INFO:tasks.ceph.osd.6.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f5af7bed640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:10.618 INFO:tasks.ceph.mds.d.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f8daba10640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:10.619 INFO:tasks.ceph.mon.b.smithi122.stderr:2023-12-17T00:00:10.620+0000 7fa93e641640 -1 Fail to open '/proc/45567/cmdline' error = (2) No such file or directory 2023-12-17T00:00:10.619 INFO:tasks.ceph.mon.b.smithi122.stderr:2023-12-17T00:00:10.620+0000 7fa93e641640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:10.619 INFO:tasks.ceph.mon.b.smithi122.stderr:2023-12-17T00:00:10.620+0000 7fa93e641640 -1 mon.b@1(peon) e1 *** Got Signal Hangup *** 2023-12-17T00:00:10.619 INFO:tasks.ceph.osd.7.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f702cf56640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:10.619 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f66974e8640 -1 Fail to open '/proc/45567/cmdline' error = (2) No such file or directory 2023-12-17T00:00:10.619 INFO:tasks.ceph.osd.5.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f66974e8640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:10.621 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f0e01b73640 -1 Fail to open '/proc/45567/cmdline' error = (2) No such file or directory 2023-12-17T00:00:10.621 INFO:tasks.ceph.osd.4.smithi122.stderr:2023-12-17T00:00:10.620+0000 7f0e01b73640 -1 received signal: Hangup from (PID: 45567) UID: 0 2023-12-17T00:00:18.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-17T00:00:18.934 DEBUG:teuthology.orchestra.run.smithi122:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-17T00:00:18.943 INFO:teuthology.orchestra.run.smithi122.stderr:error: state file /var/lib/logrotate/status is already locked 2023-12-17T00:00:18.943 INFO:teuthology.orchestra.run.smithi122.stderr:logrotate does not support parallel execution on the same set of logfiles. 2023-12-17T00:00:18.958 INFO:teuthology.orchestra.run.smithi028.stderr:error: state file /var/lib/logrotate/status is already locked 2023-12-17T00:00:18.958 INFO:teuthology.orchestra.run.smithi028.stderr:logrotate does not support parallel execution on the same set of logfiles. 2023-12-17T00:00:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 3 2023-12-17T00:00:19.050 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 189, in invoke_logrotate run.wait(procs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-12-17T00:00:59.242 INFO:teuthology.orchestra.run.smithi028.stdout:generic/239 166s 2023-12-17T00:00:59.246 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/239 2023-12-17T00:00:59.246 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:00:59.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:00:59.258 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:00:59.259 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/240) 2023-12-17T00:00:59.985 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:00:59.989 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:00:59.989 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:01:04.062 INFO:teuthology.orchestra.run.smithi028.stdout:generic/240 4s 2023-12-17T00:01:04.073 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/240 2023-12-17T00:01:04.073 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:01:04.076 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:01:04.089 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:01:04.089 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/241) 2023-12-17T00:01:04.767 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:01:04.772 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:01:04.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:18.774 INFO:teuthology.orchestra.run.smithi028.stdout:generic/241 74s 2023-12-17T00:02:18.796 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/241 2023-12-17T00:02:18.796 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:02:18.799 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:18.837 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:02:18.837 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/242) 2023-12-17T00:02:19.548 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:02:19.553 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:02:19.553 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:19.764 INFO:teuthology.orchestra.run.smithi028.stdout:generic/242 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:02:19.779 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/242 2023-12-17T00:02:19.780 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/242 2023-12-17T00:02:19.780 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:02:19.782 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:19.937 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:02:19.937 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/243) 2023-12-17T00:02:20.288 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:02:20.293 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:02:20.294 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:20.488 INFO:teuthology.orchestra.run.smithi028.stdout:generic/243 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:02:20.493 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/243 2023-12-17T00:02:20.493 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/243 2023-12-17T00:02:20.493 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:02:20.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:20.524 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:02:20.525 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/244) 2023-12-17T00:02:20.803 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:02:20.807 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:02:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:21.131 INFO:teuthology.orchestra.run.smithi028.stdout:generic/244 [not run] disk quotas not supported by this filesystem type: ceph-fuse 2023-12-17T00:02:21.136 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/244 2023-12-17T00:02:21.137 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/244 2023-12-17T00:02:21.137 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:02:21.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:21.149 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:02:21.149 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/245) 2023-12-17T00:02:21.393 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:02:21.399 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:02:21.399 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:21.833 INFO:teuthology.orchestra.run.smithi028.stdout:generic/245 0s 2023-12-17T00:02:21.868 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/245 2023-12-17T00:02:21.868 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:02:21.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:21.886 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:02:21.886 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/246) 2023-12-17T00:02:22.485 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:02:22.491 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:02:22.491 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:22.883 INFO:teuthology.orchestra.run.smithi028.stdout:generic/246 0s 2023-12-17T00:02:22.894 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/246 2023-12-17T00:02:22.894 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:02:22.896 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:02:22.906 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:02:22.907 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/247) 2023-12-17T00:02:23.478 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:02:23.482 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:02:23.483 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:26.647 INFO:teuthology.orchestra.run.smithi028.stdout:generic/247 63s 2023-12-17T00:03:26.662 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/247 2023-12-17T00:03:26.662 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:26.665 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:26.675 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:26.676 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/248) 2023-12-17T00:03:26.997 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:27.011 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:27.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:27.354 INFO:teuthology.orchestra.run.smithi028.stdout:generic/248 0s 2023-12-17T00:03:27.374 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/248 2023-12-17T00:03:27.374 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:27.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:27.399 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:27.400 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/249) 2023-12-17T00:03:27.733 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:27.745 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:27.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:30.363 INFO:teuthology.orchestra.run.smithi028.stdout:generic/249 3s 2023-12-17T00:03:30.372 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/249 2023-12-17T00:03:30.372 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:30.374 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:30.407 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:30.407 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/250) 2023-12-17T00:03:31.115 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:31.120 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:31.120 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:31.350 INFO:teuthology.orchestra.run.smithi028.stdout:generic/250 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:31.353 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/250 2023-12-17T00:03:31.354 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/250 2023-12-17T00:03:31.354 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:31.363 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:31.373 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:31.374 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/251) 2023-12-17T00:03:31.702 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:31.707 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:31.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:31.963 INFO:teuthology.orchestra.run.smithi028.stdout:generic/251 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:31.967 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/251 2023-12-17T00:03:31.967 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/251 2023-12-17T00:03:31.967 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:31.971 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:31.982 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:31.983 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/252) 2023-12-17T00:03:32.276 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:32.285 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:32.470 INFO:teuthology.orchestra.run.smithi028.stdout:generic/252 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:32.473 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/252 2023-12-17T00:03:32.473 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/252 2023-12-17T00:03:32.473 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:32.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:32.484 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:32.484 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/253) 2023-12-17T00:03:32.782 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:32.786 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:32.786 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:32.931 INFO:teuthology.orchestra.run.smithi028.stdout:generic/253 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:32.935 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/253 2023-12-17T00:03:32.935 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/253 2023-12-17T00:03:32.935 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:32.942 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:32.962 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:32.963 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/254) 2023-12-17T00:03:33.230 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:33.235 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:33.235 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:33.440 INFO:teuthology.orchestra.run.smithi028.stdout:generic/254 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:33.445 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/254 2023-12-17T00:03:33.445 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/254 2023-12-17T00:03:33.445 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:33.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:33.457 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:33.457 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/255) 2023-12-17T00:03:33.723 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:33.729 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:33.729 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:35.390 INFO:teuthology.orchestra.run.smithi028.stdout:generic/255 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-17T00:03:35.395 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/255 2023-12-17T00:03:35.395 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/255 2023-12-17T00:03:35.395 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:35.398 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:35.408 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:35.409 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/256) 2023-12-17T00:03:36.081 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:36.095 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:36.095 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:37.696 INFO:teuthology.orchestra.run.smithi028.stdout:generic/256 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:37.700 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/256 2023-12-17T00:03:37.701 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/256 2023-12-17T00:03:37.701 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:37.704 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:37.727 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:37.727 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/257) 2023-12-17T00:03:38.024 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:38.031 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:38.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:41.290 INFO:teuthology.orchestra.run.smithi028.stdout:generic/257 3s 2023-12-17T00:03:41.298 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/257 2023-12-17T00:03:41.298 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:41.300 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:41.538 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:41.539 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/258) 2023-12-17T00:03:42.096 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:42.100 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:42.375 INFO:teuthology.orchestra.run.smithi028.stdout:generic/258 [failed, exit status 1]- output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/258.out.bad) 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/258.out 2023-12-16 21:58:54.265020010 +0000 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/258.out.bad 2023-12-17 00:03:42.375233113 +0000 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,5 +1,6 @@ 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 258 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: Creating file with timestamp of Jan 1, 1960 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: Testing for negative seconds since epoch 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: -Remounting to flush cache 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: -Testing for negative seconds since epoch 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: +Timestamp wrapped: 3979351756 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: +Timestamp wrapped 2023-12-17T00:03:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: +(see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/258.full for details) 2023-12-17T00:03:42.381 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/258 2023-12-17T00:03:42.381 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/258 2023-12-17T00:03:42.381 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T00:03:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T00:03:42.397 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T00:03:42.397 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T00:03:42.398 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/259) 2023-12-17T00:03:42.700 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:42.706 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:42.856 INFO:teuthology.orchestra.run.smithi028.stdout:generic/259 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:42.860 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/259 2023-12-17T00:03:42.861 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/259 2023-12-17T00:03:42.861 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:42.884 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:42.885 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/260) 2023-12-17T00:03:43.184 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:43.188 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:43.361 INFO:teuthology.orchestra.run.smithi028.stdout:generic/260 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:43.368 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/260 2023-12-17T00:03:43.368 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/260 2023-12-17T00:03:43.368 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:43.371 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:43.382 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:43.382 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/261) 2023-12-17T00:03:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:43.738 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:43.925 INFO:teuthology.orchestra.run.smithi028.stdout:generic/261 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:43.930 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/261 2023-12-17T00:03:43.931 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/261 2023-12-17T00:03:43.931 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:43.934 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:43.946 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:43.946 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/262) 2023-12-17T00:03:44.290 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:44.295 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:44.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:44.500 INFO:teuthology.orchestra.run.smithi028.stdout:generic/262 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:03:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/262 2023-12-17T00:03:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/262 2023-12-17T00:03:44.505 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:03:44.509 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:03:44.521 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:03:44.521 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/263) 2023-12-17T00:03:44.772 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:03:44.778 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:03:44.778 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:28.067 INFO:teuthology.orchestra.run.smithi028.stdout:generic/263 103s 2023-12-17T00:05:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/263 2023-12-17T00:05:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:28.119 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:28.119 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/264) 2023-12-17T00:05:28.663 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:28.667 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:28.815 INFO:teuthology.orchestra.run.smithi028.stdout:generic/264 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:28.819 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/264 2023-12-17T00:05:28.819 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/264 2023-12-17T00:05:28.819 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:28.832 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:28.833 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/265) 2023-12-17T00:05:28.983 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:28.987 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:29.127 INFO:teuthology.orchestra.run.smithi028.stdout:generic/265 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:29.131 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/265 2023-12-17T00:05:29.131 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/265 2023-12-17T00:05:29.131 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:29.142 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:29.142 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/266) 2023-12-17T00:05:29.292 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:29.296 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:29.411 INFO:teuthology.orchestra.run.smithi028.stdout:generic/266 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:29.415 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/266 2023-12-17T00:05:29.415 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/266 2023-12-17T00:05:29.415 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:29.426 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:29.426 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/267) 2023-12-17T00:05:29.575 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:29.578 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:29.578 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:29.699 INFO:teuthology.orchestra.run.smithi028.stdout:generic/267 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:29.702 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/267 2023-12-17T00:05:29.702 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/267 2023-12-17T00:05:29.703 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:29.712 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:29.713 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/268) 2023-12-17T00:05:29.863 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:29.867 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:29.867 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:generic/268 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/268 2023-12-17T00:05:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/268 2023-12-17T00:05:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:30.009 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:30.009 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/269) 2023-12-17T00:05:30.173 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:30.177 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:30.177 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:30.278 INFO:teuthology.orchestra.run.smithi028.stdout:generic/269 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:30.282 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/269 2023-12-17T00:05:30.282 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/269 2023-12-17T00:05:30.282 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:30.285 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:30.292 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:30.293 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/270) 2023-12-17T00:05:30.440 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:30.444 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:30.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:30.558 INFO:teuthology.orchestra.run.smithi028.stdout:generic/270 [not run] disk quotas not supported by this filesystem type: ceph-fuse 2023-12-17T00:05:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/270 2023-12-17T00:05:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/270 2023-12-17T00:05:30.561 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:30.563 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:30.571 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:30.571 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/271) 2023-12-17T00:05:30.720 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:30.723 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:30.724 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:generic/271 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/271 2023-12-17T00:05:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/271 2023-12-17T00:05:30.851 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:30.853 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:30.861 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:30.861 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/272) 2023-12-17T00:05:31.010 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:31.014 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:31.014 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:31.131 INFO:teuthology.orchestra.run.smithi028.stdout:generic/272 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:31.134 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/272 2023-12-17T00:05:31.134 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/272 2023-12-17T00:05:31.134 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:31.136 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:31.144 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:31.144 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/273) 2023-12-17T00:05:31.304 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:31.308 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:31.308 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:31.412 INFO:teuthology.orchestra.run.smithi028.stdout:generic/273 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:31.416 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/273 2023-12-17T00:05:31.416 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/273 2023-12-17T00:05:31.416 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:31.418 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:31.426 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:31.426 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/274) 2023-12-17T00:05:31.576 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:31.580 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:31.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:31.683 INFO:teuthology.orchestra.run.smithi028.stdout:generic/274 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:31.686 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/274 2023-12-17T00:05:31.687 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/274 2023-12-17T00:05:31.687 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:31.689 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:31.697 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:31.697 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/275) 2023-12-17T00:05:31.849 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:31.853 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:31.853 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:31.958 INFO:teuthology.orchestra.run.smithi028.stdout:generic/275 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:31.962 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/275 2023-12-17T00:05:31.962 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/275 2023-12-17T00:05:31.962 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:31.964 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:31.971 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:31.972 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/276) 2023-12-17T00:05:32.119 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:32.123 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:32.123 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:32.247 INFO:teuthology.orchestra.run.smithi028.stdout:generic/276 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:32.251 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/276 2023-12-17T00:05:32.251 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/276 2023-12-17T00:05:32.251 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:32.253 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:32.261 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:32.261 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/277) 2023-12-17T00:05:32.409 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:32.413 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:32.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:32.525 INFO:teuthology.orchestra.run.smithi028.stdout:generic/277 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:32.529 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/277 2023-12-17T00:05:32.529 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/277 2023-12-17T00:05:32.529 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:32.531 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:32.539 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:32.539 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/278) 2023-12-17T00:05:32.687 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:32.691 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:32.691 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:32.831 INFO:teuthology.orchestra.run.smithi028.stdout:generic/278 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:32.834 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/278 2023-12-17T00:05:32.834 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/278 2023-12-17T00:05:32.835 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:32.836 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:32.844 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:32.844 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/279) 2023-12-17T00:05:32.995 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:32.999 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:32.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:33.119 INFO:teuthology.orchestra.run.smithi028.stdout:generic/279 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:33.122 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/279 2023-12-17T00:05:33.122 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/279 2023-12-17T00:05:33.122 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:33.166 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:33.224 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:33.224 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/280) 2023-12-17T00:05:33.389 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:33.393 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:33.394 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:33.595 INFO:teuthology.orchestra.run.smithi028.stdout:generic/280 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:33.601 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/280 2023-12-17T00:05:33.602 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/280 2023-12-17T00:05:33.602 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:33.605 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:33.616 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:33.616 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/281) 2023-12-17T00:05:33.783 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:33.787 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:33.787 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:33.903 INFO:teuthology.orchestra.run.smithi028.stdout:generic/281 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:33.907 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/281 2023-12-17T00:05:33.907 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/281 2023-12-17T00:05:33.907 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:33.909 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:33.917 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:33.917 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/282) 2023-12-17T00:05:34.067 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:34.070 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:34.192 INFO:teuthology.orchestra.run.smithi028.stdout:generic/282 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:34.195 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/282 2023-12-17T00:05:34.195 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/282 2023-12-17T00:05:34.195 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:34.197 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:34.205 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:34.206 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/283) 2023-12-17T00:05:34.358 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:34.362 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:34.362 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:34.480 INFO:teuthology.orchestra.run.smithi028.stdout:generic/283 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:34.483 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/283 2023-12-17T00:05:34.483 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/283 2023-12-17T00:05:34.483 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:34.485 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:34.494 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:34.494 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/284) 2023-12-17T00:05:34.646 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:34.649 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:34.650 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:34.751 INFO:teuthology.orchestra.run.smithi028.stdout:generic/284 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:05:34.755 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/284 2023-12-17T00:05:34.755 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/284 2023-12-17T00:05:34.755 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:34.757 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:34.765 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:34.766 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/285) 2023-12-17T00:05:34.915 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:34.919 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:34.919 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:37.756 INFO:teuthology.orchestra.run.smithi028.stdout:generic/285 3s 2023-12-17T00:05:37.795 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/285 2023-12-17T00:05:37.795 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:05:37.798 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:05:37.810 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:05:37.811 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/286) 2023-12-17T00:05:38.381 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:05:38.385 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:05:38.385 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:18.985 INFO:teuthology.orchestra.run.smithi028.stdout:generic/286 40s 2023-12-17T00:06:18.991 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/286 2023-12-17T00:06:18.991 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:18.993 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:19.235 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:19.235 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/287) 2023-12-17T00:06:19.694 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:19.710 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:19.710 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:19.932 INFO:teuthology.orchestra.run.smithi028.stdout:generic/287 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:19.940 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/287 2023-12-17T00:06:19.940 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/287 2023-12-17T00:06:19.940 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:19.943 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:19.958 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:19.958 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/288) 2023-12-17T00:06:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:20.128 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:generic/288 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:20.246 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/288 2023-12-17T00:06:20.246 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/288 2023-12-17T00:06:20.246 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:20.249 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:20.257 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:20.257 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/289) 2023-12-17T00:06:20.411 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:20.415 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:20.415 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:20.744 INFO:teuthology.orchestra.run.smithi028.stdout:generic/289 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:20.748 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/289 2023-12-17T00:06:20.748 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/289 2023-12-17T00:06:20.748 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:20.759 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:20.760 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/290) 2023-12-17T00:06:20.913 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:20.916 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:20.917 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:21.032 INFO:teuthology.orchestra.run.smithi028.stdout:generic/290 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:21.035 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/290 2023-12-17T00:06:21.035 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/290 2023-12-17T00:06:21.036 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:21.038 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:21.046 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:21.046 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/291) 2023-12-17T00:06:21.201 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:21.205 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:21.205 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:21.308 INFO:teuthology.orchestra.run.smithi028.stdout:generic/291 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:21.312 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/291 2023-12-17T00:06:21.312 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/291 2023-12-17T00:06:21.312 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:21.314 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:21.323 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:21.323 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/292) 2023-12-17T00:06:21.474 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:21.478 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:21.478 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:21.624 INFO:teuthology.orchestra.run.smithi028.stdout:generic/292 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:21.627 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/292 2023-12-17T00:06:21.628 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/292 2023-12-17T00:06:21.628 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:21.630 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:21.639 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:21.639 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/293) 2023-12-17T00:06:21.793 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:21.797 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:21.797 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:22.194 INFO:teuthology.orchestra.run.smithi028.stdout:generic/293 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:22.198 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/293 2023-12-17T00:06:22.198 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/293 2023-12-17T00:06:22.198 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:22.200 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:22.208 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:22.209 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/294) 2023-12-17T00:06:22.363 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:22.367 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:22.367 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:22.527 INFO:teuthology.orchestra.run.smithi028.stdout:generic/294 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:22.531 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/294 2023-12-17T00:06:22.531 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/294 2023-12-17T00:06:22.531 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:22.533 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:22.542 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:22.542 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/295) 2023-12-17T00:06:22.697 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:22.701 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:22.701 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:23.094 INFO:teuthology.orchestra.run.smithi028.stdout:generic/295 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/295 2023-12-17T00:06:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/295 2023-12-17T00:06:23.098 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:23.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:23.109 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:23.109 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/296) 2023-12-17T00:06:23.266 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:23.270 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:23.270 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:23.503 INFO:teuthology.orchestra.run.smithi028.stdout:generic/296 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:23.507 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/296 2023-12-17T00:06:23.507 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/296 2023-12-17T00:06:23.507 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:23.509 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:23.518 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:23.519 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/297) 2023-12-17T00:06:23.680 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:23.684 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:23.684 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:23.871 INFO:teuthology.orchestra.run.smithi028.stdout:generic/297 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:23.876 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/297 2023-12-17T00:06:23.876 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/297 2023-12-17T00:06:23.876 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:23.879 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:23.887 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:23.887 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/298) 2023-12-17T00:06:24.050 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:24.054 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:24.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:24.162 INFO:teuthology.orchestra.run.smithi028.stdout:generic/298 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:24.166 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/298 2023-12-17T00:06:24.166 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/298 2023-12-17T00:06:24.166 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:24.168 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:24.177 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:24.177 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/299) 2023-12-17T00:06:24.341 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:24.345 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:24.497 INFO:teuthology.orchestra.run.smithi028.stdout:generic/299 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:24.501 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/299 2023-12-17T00:06:24.501 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/299 2023-12-17T00:06:24.501 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:24.511 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:24.512 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/300) 2023-12-17T00:06:24.676 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:24.680 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:24.681 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:24.969 INFO:teuthology.orchestra.run.smithi028.stdout:generic/300 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:24.974 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/300 2023-12-17T00:06:24.974 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/300 2023-12-17T00:06:24.974 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:24.985 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:24.985 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/301) 2023-12-17T00:06:25.135 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:25.139 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:25.441 INFO:teuthology.orchestra.run.smithi028.stdout:generic/301 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:25.446 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/301 2023-12-17T00:06:25.446 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/301 2023-12-17T00:06:25.446 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:25.493 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:25.493 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/302) 2023-12-17T00:06:25.677 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:25.681 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:25.681 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:25.797 INFO:teuthology.orchestra.run.smithi028.stdout:generic/302 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:25.802 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/302 2023-12-17T00:06:25.802 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/302 2023-12-17T00:06:25.802 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:25.812 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:25.813 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/303) 2023-12-17T00:06:25.965 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:25.969 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:26.159 INFO:teuthology.orchestra.run.smithi028.stdout:generic/303 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-17T00:06:26.162 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/303 2023-12-17T00:06:26.162 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/303 2023-12-17T00:06:26.162 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:26.165 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:26.173 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:26.173 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/304) 2023-12-17T00:06:26.796 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:26.801 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:26.801 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:26.965 INFO:teuthology.orchestra.run.smithi028.stdout:generic/304 [not run] Dedupe not supported by test filesystem type: ceph-fuse 2023-12-17T00:06:26.968 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/304 2023-12-17T00:06:26.968 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/304 2023-12-17T00:06:26.968 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:26.970 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:26.978 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:26.978 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/305) 2023-12-17T00:06:27.578 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:27.582 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:27.582 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:27.746 INFO:teuthology.orchestra.run.smithi028.stdout:generic/305 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:27.750 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/305 2023-12-17T00:06:27.750 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/305 2023-12-17T00:06:27.750 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:27.752 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:27.761 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:27.762 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/306) 2023-12-17T00:06:27.916 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:27.920 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:28.026 INFO:teuthology.orchestra.run.smithi028.stdout:generic/306 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:28.029 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/306 2023-12-17T00:06:28.029 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/306 2023-12-17T00:06:28.030 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:28.040 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:28.040 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/307) 2023-12-17T00:06:28.191 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:28.195 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:28.344 INFO:teuthology.orchestra.run.smithi028.stdout:generic/307 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:06:28.348 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/307 2023-12-17T00:06:28.348 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/307 2023-12-17T00:06:28.348 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:28.359 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:28.359 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/308) 2023-12-17T00:06:28.511 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:28.515 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:28.667 INFO:teuthology.orchestra.run.smithi028.stdout:generic/308 0s 2023-12-17T00:06:28.673 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/308 2023-12-17T00:06:28.673 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:28.683 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:28.684 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/309) 2023-12-17T00:06:28.839 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:28.843 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:generic/309 2s 2023-12-17T00:06:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/309 2023-12-17T00:06:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:06:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:06:30.212 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:06:30.212 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/310) 2023-12-17T00:06:30.686 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:06:30.689 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:06:30.690 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:07:54.524 INFO:teuthology.orchestra.run.smithi028.stdout:generic/310 84s 2023-12-17T00:07:54.529 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/310 2023-12-17T00:07:54.530 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:07:54.532 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:07:54.577 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:07:54.577 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/311) 2023-12-17T00:07:54.736 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:07:54.740 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:07:54.740 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:07:55.077 INFO:teuthology.orchestra.run.smithi028.stdout:generic/311 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:07:55.167 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/311 2023-12-17T00:07:55.167 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/311 2023-12-17T00:07:55.167 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:07:55.285 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:07:55.303 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:07:55.304 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/312) 2023-12-17T00:07:55.694 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:07:55.698 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:07:55.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:07:55.860 INFO:teuthology.orchestra.run.smithi028.stdout:generic/312 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:07:55.872 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/312 2023-12-17T00:07:55.872 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/312 2023-12-17T00:07:55.872 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:07:55.885 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:07:55.934 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:07:55.934 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/313) 2023-12-17T00:07:56.327 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:07:56.331 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:07:56.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:00.613 INFO:teuthology.orchestra.run.smithi028.stdout:generic/313 4s 2023-12-17T00:08:00.650 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/313 2023-12-17T00:08:00.650 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:00.654 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:00.666 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:00.666 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/314) 2023-12-17T00:08:00.866 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:00.870 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:03.523 INFO:teuthology.orchestra.run.smithi028.stdout:generic/314 3s 2023-12-17T00:08:03.530 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/314 2023-12-17T00:08:03.530 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:03.533 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:03.544 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:03.544 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/315) 2023-12-17T00:08:03.707 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:03.711 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:03.711 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:05.106 INFO:teuthology.orchestra.run.smithi028.stdout:generic/315 2s 2023-12-17T00:08:05.113 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/315 2023-12-17T00:08:05.113 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:05.276 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:05.276 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/316) 2023-12-17T00:08:05.741 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:05.745 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:06.867 INFO:teuthology.orchestra.run.smithi028.stdout:generic/316 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-17T00:08:06.871 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/316 2023-12-17T00:08:06.871 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/316 2023-12-17T00:08:06.871 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:06.883 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:06.883 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/317) 2023-12-17T00:08:07.466 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:07.470 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:07.470 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:07.571 INFO:teuthology.orchestra.run.smithi028.stdout:generic/317 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:08:07.574 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/317 2023-12-17T00:08:07.574 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/317 2023-12-17T00:08:07.575 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:07.577 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:07.585 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:07.585 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/318) 2023-12-17T00:08:07.734 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:07.738 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:07.738 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:07.843 INFO:teuthology.orchestra.run.smithi028.stdout:generic/318 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:08:07.848 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/318 2023-12-17T00:08:07.848 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/318 2023-12-17T00:08:07.848 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:07.856 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:07.864 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:07.865 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/319) 2023-12-17T00:08:08.016 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:08.020 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:08.020 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:08.158 INFO:teuthology.orchestra.run.smithi028.stdout:generic/319 [not run] ACLs not supported by this filesystem type: ceph-fuse 2023-12-17T00:08:08.162 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/319 2023-12-17T00:08:08.162 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/319 2023-12-17T00:08:08.162 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:08.173 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:08.173 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/320) 2023-12-17T00:08:08.323 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:08.327 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:08.327 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:08.428 INFO:teuthology.orchestra.run.smithi028.stdout:generic/320 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:08:08.432 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/320 2023-12-17T00:08:08.432 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/320 2023-12-17T00:08:08.432 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:08.434 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:08.442 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:08.442 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/321) 2023-12-17T00:08:08.590 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:08.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:08.715 INFO:teuthology.orchestra.run.smithi028.stdout:generic/321 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:08:08.719 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/321 2023-12-17T00:08:08.719 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/321 2023-12-17T00:08:08.719 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:08.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:08.728 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:08.728 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/322) 2023-12-17T00:08:08.881 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:08.884 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:08.885 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:08.995 INFO:teuthology.orchestra.run.smithi028.stdout:generic/322 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:08:08.999 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/322 2023-12-17T00:08:08.999 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/322 2023-12-17T00:08:08.999 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:08:09.001 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:08:09.010 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:08:09.010 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/323) 2023-12-17T00:08:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:08:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:08:09.179 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:10.418 INFO:teuthology.orchestra.run.smithi028.stdout:generic/323 121s 2023-12-17T00:10:10.551 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/323 2023-12-17T00:10:10.551 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:10.557 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:10.573 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:10.574 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/324) 2023-12-17T00:10:11.088 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:11.092 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:11.092 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:11.222 INFO:teuthology.orchestra.run.smithi028.stdout:generic/324 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:11.226 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/324 2023-12-17T00:10:11.226 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/324 2023-12-17T00:10:11.226 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:11.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:11.236 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:11.237 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/325) 2023-12-17T00:10:11.389 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:11.393 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:11.393 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:11.524 INFO:teuthology.orchestra.run.smithi028.stdout:generic/325 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:11.527 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/325 2023-12-17T00:10:11.528 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/325 2023-12-17T00:10:11.528 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:11.530 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:11.538 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:11.538 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/326) 2023-12-17T00:10:11.688 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:11.692 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:11.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:11.796 INFO:teuthology.orchestra.run.smithi028.stdout:generic/326 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:11.799 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/326 2023-12-17T00:10:11.799 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/326 2023-12-17T00:10:11.799 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:11.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:11.810 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:11.810 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/327) 2023-12-17T00:10:11.965 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:11.968 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:11.969 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:12.079 INFO:teuthology.orchestra.run.smithi028.stdout:generic/327 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:12.082 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/327 2023-12-17T00:10:12.082 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/327 2023-12-17T00:10:12.082 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:12.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:12.092 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:12.093 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/328) 2023-12-17T00:10:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:12.248 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:12.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:12.349 INFO:teuthology.orchestra.run.smithi028.stdout:generic/328 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:12.353 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/328 2023-12-17T00:10:12.353 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/328 2023-12-17T00:10:12.353 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:12.355 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:12.363 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:12.363 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/329) 2023-12-17T00:10:12.512 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:12.516 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:12.516 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:12.644 INFO:teuthology.orchestra.run.smithi028.stdout:generic/329 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:12.648 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/329 2023-12-17T00:10:12.648 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/329 2023-12-17T00:10:12.648 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:12.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:12.659 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:12.660 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/330) 2023-12-17T00:10:12.810 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:12.814 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:12.917 INFO:teuthology.orchestra.run.smithi028.stdout:generic/330 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:12.920 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/330 2023-12-17T00:10:12.920 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/330 2023-12-17T00:10:12.921 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:12.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:12.931 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:12.931 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/331) 2023-12-17T00:10:13.082 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:13.086 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:13.086 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:13.200 INFO:teuthology.orchestra.run.smithi028.stdout:generic/331 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:13.204 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/331 2023-12-17T00:10:13.204 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/331 2023-12-17T00:10:13.204 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:13.206 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:13.214 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:13.214 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/332) 2023-12-17T00:10:13.363 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:13.367 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:13.367 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:13.471 INFO:teuthology.orchestra.run.smithi028.stdout:generic/332 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:13.474 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/332 2023-12-17T00:10:13.475 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/332 2023-12-17T00:10:13.475 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:13.477 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:13.485 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:13.485 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/333) 2023-12-17T00:10:13.633 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:13.637 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:13.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:13.739 INFO:teuthology.orchestra.run.smithi028.stdout:generic/333 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:13.743 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/333 2023-12-17T00:10:13.743 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/333 2023-12-17T00:10:13.743 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:13.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:13.754 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:13.754 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/334) 2023-12-17T00:10:13.912 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:13.916 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:14.017 INFO:teuthology.orchestra.run.smithi028.stdout:generic/334 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:14.020 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/334 2023-12-17T00:10:14.020 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/334 2023-12-17T00:10:14.020 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:14.031 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:14.031 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/335) 2023-12-17T00:10:14.182 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:14.186 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:14.186 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:14.300 INFO:teuthology.orchestra.run.smithi028.stdout:generic/335 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:14.304 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/335 2023-12-17T00:10:14.304 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/335 2023-12-17T00:10:14.304 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:14.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:14.314 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:14.314 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/336) 2023-12-17T00:10:14.465 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:14.469 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:14.469 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:14.593 INFO:teuthology.orchestra.run.smithi028.stdout:generic/336 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:14.597 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/336 2023-12-17T00:10:14.597 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/336 2023-12-17T00:10:14.597 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:14.599 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:14.609 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:14.610 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/337) 2023-12-17T00:10:14.764 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:14.768 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:14.769 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:14.871 INFO:teuthology.orchestra.run.smithi028.stdout:generic/337 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:14.874 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/337 2023-12-17T00:10:14.875 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/337 2023-12-17T00:10:14.875 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:14.885 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:14.885 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/338) 2023-12-17T00:10:15.036 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:15.040 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:15.176 INFO:teuthology.orchestra.run.smithi028.stdout:generic/338 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:15.180 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/338 2023-12-17T00:10:15.181 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/338 2023-12-17T00:10:15.181 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:15.191 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:15.191 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/339) 2023-12-17T00:10:15.352 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:15.356 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:15.456 INFO:teuthology.orchestra.run.smithi028.stdout:generic/339 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:15.460 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/339 2023-12-17T00:10:15.460 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/339 2023-12-17T00:10:15.460 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:15.470 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:15.470 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/340) 2023-12-17T00:10:15.622 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:15.626 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:15.727 INFO:teuthology.orchestra.run.smithi028.stdout:generic/340 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:15.730 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/340 2023-12-17T00:10:15.731 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/340 2023-12-17T00:10:15.731 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:15.741 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:15.741 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/341) 2023-12-17T00:10:15.890 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:15.894 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:16.170 INFO:teuthology.orchestra.run.smithi028.stdout:generic/341 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:16.174 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/341 2023-12-17T00:10:16.174 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/341 2023-12-17T00:10:16.174 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:16.187 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:16.187 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/342) 2023-12-17T00:10:16.355 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:16.360 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:16.360 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:16.541 INFO:teuthology.orchestra.run.smithi028.stdout:generic/342 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:16.544 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/342 2023-12-17T00:10:16.545 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/342 2023-12-17T00:10:16.545 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:16.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:16.555 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:16.555 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/343) 2023-12-17T00:10:16.708 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:16.712 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:16.713 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:17.019 INFO:teuthology.orchestra.run.smithi028.stdout:generic/343 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/343 2023-12-17T00:10:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/343 2023-12-17T00:10:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:17.026 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:17.035 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:17.036 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/344) 2023-12-17T00:10:17.201 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:17.206 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:17.206 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:17.341 INFO:teuthology.orchestra.run.smithi028.stdout:generic/344 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/344 2023-12-17T00:10:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/344 2023-12-17T00:10:17.345 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:17.347 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:17.356 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:17.356 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/345) 2023-12-17T00:10:17.519 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:17.524 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:17.524 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:17.656 INFO:teuthology.orchestra.run.smithi028.stdout:generic/345 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:17.661 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/345 2023-12-17T00:10:17.661 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/345 2023-12-17T00:10:17.661 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:17.663 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:17.674 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:17.674 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/346) 2023-12-17T00:10:17.834 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:17.837 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:17.837 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:17.944 INFO:teuthology.orchestra.run.smithi028.stdout:generic/346 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:17.948 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/346 2023-12-17T00:10:17.948 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/346 2023-12-17T00:10:17.948 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:17.958 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:17.958 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/347) 2023-12-17T00:10:18.109 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:18.112 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:18.113 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:18.318 INFO:teuthology.orchestra.run.smithi028.stdout:generic/347 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:18.322 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/347 2023-12-17T00:10:18.322 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/347 2023-12-17T00:10:18.323 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:18.333 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:18.333 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/348) 2023-12-17T00:10:18.489 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:18.491 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:18.633 INFO:teuthology.orchestra.run.smithi028.stdout:generic/348 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:18.638 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/348 2023-12-17T00:10:18.638 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/348 2023-12-17T00:10:18.638 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:18.640 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:18.648 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:18.648 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/349) 2023-12-17T00:10:18.803 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:18.807 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:20.935 INFO:teuthology.orchestra.run.smithi028.stdout:generic/349 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-12-17T00:10:20.942 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/349 2023-12-17T00:10:20.942 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/349 2023-12-17T00:10:20.942 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:20.946 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:20.959 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:20.959 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/350) 2023-12-17T00:10:21.162 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:21.167 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:21.167 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:23.140 INFO:teuthology.orchestra.run.smithi028.stdout:generic/350 2s 2023-12-17T00:10:23.147 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/350 2023-12-17T00:10:23.148 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:23.151 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:23.161 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:23.162 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/351) 2023-12-17T00:10:23.358 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:23.363 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:23.363 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:24.579 INFO:teuthology.orchestra.run.smithi028.stdout:generic/351 [not run] xfs_io finsert failed (old kernel/wrong fs?) 2023-12-17T00:10:24.585 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/351 2023-12-17T00:10:24.585 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/351 2023-12-17T00:10:24.585 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:24.603 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:24.603 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/352) 2023-12-17T00:10:25.184 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:25.189 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:25.302 INFO:teuthology.orchestra.run.smithi028.stdout:generic/352 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:25.306 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/352 2023-12-17T00:10:25.306 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/352 2023-12-17T00:10:25.306 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:25.317 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:25.317 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/353) 2023-12-17T00:10:25.471 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:25.475 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:25.581 INFO:teuthology.orchestra.run.smithi028.stdout:generic/353 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:25.585 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/353 2023-12-17T00:10:25.585 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/353 2023-12-17T00:10:25.585 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:25.587 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:25.596 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:25.597 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/354) 2023-12-17T00:10:25.753 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:25.757 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:25.858 INFO:teuthology.orchestra.run.smithi028.stdout:generic/354 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:25.861 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/354 2023-12-17T00:10:25.861 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/354 2023-12-17T00:10:25.862 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:25.872 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:25.872 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/355) 2023-12-17T00:10:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:26.031 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:26.564 INFO:teuthology.orchestra.run.smithi028.stdout:generic/355 0s 2023-12-17T00:10:26.568 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/355 2023-12-17T00:10:26.568 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:26.578 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:26.579 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/356) 2023-12-17T00:10:27.102 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:27.106 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:27.106 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:27.209 INFO:teuthology.orchestra.run.smithi028.stdout:generic/356 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:27.212 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/356 2023-12-17T00:10:27.212 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/356 2023-12-17T00:10:27.213 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:27.215 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:27.223 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:27.223 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/357) 2023-12-17T00:10:27.374 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:27.378 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:27.378 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:27.479 INFO:teuthology.orchestra.run.smithi028.stdout:generic/357 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:27.483 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/357 2023-12-17T00:10:27.483 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/357 2023-12-17T00:10:27.483 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:27.493 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:27.494 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/358) 2023-12-17T00:10:27.642 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:27.645 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:27.646 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:27.748 INFO:teuthology.orchestra.run.smithi028.stdout:generic/358 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:27.751 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/358 2023-12-17T00:10:27.751 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/358 2023-12-17T00:10:27.751 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:27.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:27.762 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:27.762 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/359) 2023-12-17T00:10:27.913 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:27.916 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:28.018 INFO:teuthology.orchestra.run.smithi028.stdout:generic/359 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:28.021 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/359 2023-12-17T00:10:28.021 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/359 2023-12-17T00:10:28.022 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:28.032 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:28.032 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/360) 2023-12-17T00:10:28.181 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:28.184 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:28.387 INFO:teuthology.orchestra.run.smithi028.stdout:generic/360 0s 2023-12-17T00:10:28.393 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/360 2023-12-17T00:10:28.393 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:28.405 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:28.405 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/361) 2023-12-17T00:10:28.558 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:28.562 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:28.700 INFO:teuthology.orchestra.run.smithi028.stdout:generic/361 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:28.704 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/361 2023-12-17T00:10:28.704 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/361 2023-12-17T00:10:28.704 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:28.716 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:28.716 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/362) 2023-12-17T00:10:28.871 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:28.875 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:28.976 INFO:teuthology.orchestra.run.smithi028.stdout:generic/362 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:28.979 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/362 2023-12-17T00:10:28.979 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/362 2023-12-17T00:10:28.979 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:28.989 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:28.990 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/363) 2023-12-17T00:10:29.140 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:29.144 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:29.243 INFO:teuthology.orchestra.run.smithi028.stdout:generic/363 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:29.247 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/363 2023-12-17T00:10:29.247 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/363 2023-12-17T00:10:29.247 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:29.249 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:29.257 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:29.258 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/364) 2023-12-17T00:10:29.417 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:29.421 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:29.522 INFO:teuthology.orchestra.run.smithi028.stdout:generic/364 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:29.526 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/364 2023-12-17T00:10:29.526 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/364 2023-12-17T00:10:29.526 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:29.528 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:29.536 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:29.536 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/365) 2023-12-17T00:10:29.691 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:29.696 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:29.804 INFO:teuthology.orchestra.run.smithi028.stdout:generic/365 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:29.820 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/365 2023-12-17T00:10:29.820 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/365 2023-12-17T00:10:29.820 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:29.820 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:29.821 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:29.821 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/366) 2023-12-17T00:10:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:29.975 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:generic/366 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:30.080 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/366 2023-12-17T00:10:30.080 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/366 2023-12-17T00:10:30.080 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:30.090 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:30.091 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/367) 2023-12-17T00:10:30.243 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:30.247 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:30.247 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:30.347 INFO:teuthology.orchestra.run.smithi028.stdout:generic/367 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:30.351 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/367 2023-12-17T00:10:30.351 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/367 2023-12-17T00:10:30.351 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:30.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:30.361 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:30.361 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/368) 2023-12-17T00:10:30.516 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:30.520 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:30.520 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:30.621 INFO:teuthology.orchestra.run.smithi028.stdout:generic/368 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:30.625 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/368 2023-12-17T00:10:30.625 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/368 2023-12-17T00:10:30.625 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:30.627 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:30.635 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:30.635 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/369) 2023-12-17T00:10:30.784 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:30.788 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:30.788 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:30.888 INFO:teuthology.orchestra.run.smithi028.stdout:generic/369 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/369 2023-12-17T00:10:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/369 2023-12-17T00:10:30.891 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:30.894 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:30.902 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:30.903 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/370) 2023-12-17T00:10:31.057 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:31.061 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:31.061 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:31.163 INFO:teuthology.orchestra.run.smithi028.stdout:generic/370 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:31.167 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/370 2023-12-17T00:10:31.167 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/370 2023-12-17T00:10:31.167 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:31.177 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:31.177 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/371) 2023-12-17T00:10:31.328 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:31.332 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:31.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:31.437 INFO:teuthology.orchestra.run.smithi028.stdout:generic/371 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:31.441 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/371 2023-12-17T00:10:31.441 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/371 2023-12-17T00:10:31.441 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:31.443 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:31.451 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:31.451 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/372) 2023-12-17T00:10:31.601 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:31.604 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:31.605 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:31.705 INFO:teuthology.orchestra.run.smithi028.stdout:generic/372 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:31.709 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/372 2023-12-17T00:10:31.709 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/372 2023-12-17T00:10:31.709 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:31.711 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:31.719 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:31.719 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/373) 2023-12-17T00:10:31.868 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:31.872 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:31.872 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:31.975 INFO:teuthology.orchestra.run.smithi028.stdout:generic/373 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:31.978 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/373 2023-12-17T00:10:31.979 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/373 2023-12-17T00:10:31.979 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:31.981 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:31.989 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:31.989 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/374) 2023-12-17T00:10:32.139 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:32.143 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:32.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:32.250 INFO:teuthology.orchestra.run.smithi028.stdout:generic/374 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:32.254 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/374 2023-12-17T00:10:32.254 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/374 2023-12-17T00:10:32.254 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:32.264 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:32.265 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/375) 2023-12-17T00:10:32.414 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:32.417 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:32.417 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:32.566 INFO:teuthology.orchestra.run.smithi028.stdout:generic/375 [not run] ACLs not supported by this filesystem type: ceph-fuse 2023-12-17T00:10:32.569 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/375 2023-12-17T00:10:32.569 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/375 2023-12-17T00:10:32.569 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:32.571 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:32.579 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:32.580 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/376) 2023-12-17T00:10:32.728 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:32.732 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:32.732 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:32.857 INFO:teuthology.orchestra.run.smithi028.stdout:generic/376 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:32.860 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/376 2023-12-17T00:10:32.860 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/376 2023-12-17T00:10:32.860 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:32.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:32.871 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:32.871 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/377) 2023-12-17T00:10:33.023 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:33.032 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:33.133 INFO:teuthology.orchestra.run.smithi028.stdout:generic/377 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:33.137 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/377 2023-12-17T00:10:33.137 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/377 2023-12-17T00:10:33.137 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:33.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:33.147 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:33.148 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/378) 2023-12-17T00:10:33.301 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:33.305 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:33.306 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:33.500 INFO:teuthology.orchestra.run.smithi028.stdout:generic/378 0s 2023-12-17T00:10:33.504 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/378 2023-12-17T00:10:33.505 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:33.507 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:33.515 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:33.515 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/379) 2023-12-17T00:10:34.026 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:34.030 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:34.134 INFO:teuthology.orchestra.run.smithi028.stdout:generic/379 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:34.138 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/379 2023-12-17T00:10:34.138 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/379 2023-12-17T00:10:34.138 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:34.140 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:34.148 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:34.148 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/380) 2023-12-17T00:10:34.299 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:34.303 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:34.304 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:34.406 INFO:teuthology.orchestra.run.smithi028.stdout:generic/380 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:34.409 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/380 2023-12-17T00:10:34.410 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/380 2023-12-17T00:10:34.410 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:34.412 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:34.420 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:34.420 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/381) 2023-12-17T00:10:34.571 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:34.575 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:34.575 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:34.679 INFO:teuthology.orchestra.run.smithi028.stdout:generic/381 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:34.682 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/381 2023-12-17T00:10:34.682 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/381 2023-12-17T00:10:34.682 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:34.684 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:34.692 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:34.692 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/382) 2023-12-17T00:10:34.839 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:34.843 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:34.843 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:34.946 INFO:teuthology.orchestra.run.smithi028.stdout:generic/382 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:34.950 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/382 2023-12-17T00:10:34.950 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/382 2023-12-17T00:10:34.950 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:34.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:34.960 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:34.960 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/383) 2023-12-17T00:10:35.108 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:35.112 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:35.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:35.214 INFO:teuthology.orchestra.run.smithi028.stdout:generic/383 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/383 2023-12-17T00:10:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/383 2023-12-17T00:10:35.218 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:35.220 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:35.228 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:35.228 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/384) 2023-12-17T00:10:35.382 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:35.386 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:35.492 INFO:teuthology.orchestra.run.smithi028.stdout:generic/384 [not run] disk quotas not supported by this filesystem type: ceph-fuse 2023-12-17T00:10:35.496 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/384 2023-12-17T00:10:35.496 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/384 2023-12-17T00:10:35.496 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:35.498 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:35.506 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:35.506 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/385) 2023-12-17T00:10:35.653 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:35.657 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:35.657 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:35.760 INFO:teuthology.orchestra.run.smithi028.stdout:generic/385 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:35.764 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/385 2023-12-17T00:10:35.764 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/385 2023-12-17T00:10:35.764 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:35.767 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:35.775 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:35.775 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/386) 2023-12-17T00:10:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:35.934 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:35.934 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:36.036 INFO:teuthology.orchestra.run.smithi028.stdout:generic/386 [not run] disk quotas not supported by this filesystem type: ceph-fuse 2023-12-17T00:10:36.039 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/386 2023-12-17T00:10:36.039 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/386 2023-12-17T00:10:36.039 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:36.042 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:36.050 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:36.050 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/387) 2023-12-17T00:10:36.208 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:36.212 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:36.212 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:36.314 INFO:teuthology.orchestra.run.smithi028.stdout:generic/387 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:36.318 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/387 2023-12-17T00:10:36.318 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/387 2023-12-17T00:10:36.318 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:36.320 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:36.328 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:36.328 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/388) 2023-12-17T00:10:36.477 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:36.481 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:36.481 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:36.597 INFO:teuthology.orchestra.run.smithi028.stdout:generic/388 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:36.600 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/388 2023-12-17T00:10:36.600 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/388 2023-12-17T00:10:36.600 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:36.602 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:36.610 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:36.611 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/389) 2023-12-17T00:10:36.761 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:36.765 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:36.765 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:36.874 INFO:teuthology.orchestra.run.smithi028.stdout:generic/389 [not run] O_TMPFILE is not supported 2023-12-17T00:10:36.877 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/389 2023-12-17T00:10:36.878 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/389 2023-12-17T00:10:36.878 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:36.888 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:36.888 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/390) 2023-12-17T00:10:37.036 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:37.040 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:37.041 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:37.154 INFO:teuthology.orchestra.run.smithi028.stdout:generic/390 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:10:37.158 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/390 2023-12-17T00:10:37.158 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/390 2023-12-17T00:10:37.158 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:10:37.160 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:10:37.167 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:10:37.168 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/391) 2023-12-17T00:10:37.318 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:10:37.322 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:10:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:24.274 INFO:teuthology.orchestra.run.smithi028.stdout:generic/391 47s 2023-12-17T00:11:24.413 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/391 2023-12-17T00:11:24.413 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:24.523 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:24.523 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/392) 2023-12-17T00:11:25.974 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:25.978 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:26.167 INFO:teuthology.orchestra.run.smithi028.stdout:generic/392 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:26.171 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/392 2023-12-17T00:11:26.171 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/392 2023-12-17T00:11:26.171 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:26.174 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:26.203 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:26.203 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/393) 2023-12-17T00:11:26.462 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:26.467 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:26.609 INFO:teuthology.orchestra.run.smithi028.stdout:generic/393 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:26.612 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/393 2023-12-17T00:11:26.612 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/393 2023-12-17T00:11:26.612 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:26.615 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:26.623 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:26.624 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/394) 2023-12-17T00:11:26.928 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:26.934 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:26.934 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:27.179 INFO:teuthology.orchestra.run.smithi028.stdout:generic/394 1s 2023-12-17T00:11:27.188 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/394 2023-12-17T00:11:27.188 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:27.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:27.210 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:27.210 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/395) 2023-12-17T00:11:27.815 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:27.827 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:27.827 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:28.035 INFO:teuthology.orchestra.run.smithi028.stdout:generic/395 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:28.040 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/395 2023-12-17T00:11:28.040 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/395 2023-12-17T00:11:28.040 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:28.061 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:28.062 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/396) 2023-12-17T00:11:28.319 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:28.323 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:28.515 INFO:teuthology.orchestra.run.smithi028.stdout:generic/396 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:28.519 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/396 2023-12-17T00:11:28.519 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/396 2023-12-17T00:11:28.519 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:28.534 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:28.534 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/397) 2023-12-17T00:11:28.769 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:28.773 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:28.960 INFO:teuthology.orchestra.run.smithi028.stdout:generic/397 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:28.964 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/397 2023-12-17T00:11:28.964 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/397 2023-12-17T00:11:28.964 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:28.974 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:28.974 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/398) 2023-12-17T00:11:29.123 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:29.126 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:29.257 INFO:teuthology.orchestra.run.smithi028.stdout:generic/398 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:29.260 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/398 2023-12-17T00:11:29.260 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/398 2023-12-17T00:11:29.261 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:29.270 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:29.271 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/399) 2023-12-17T00:11:29.422 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:29.426 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:29.557 INFO:teuthology.orchestra.run.smithi028.stdout:generic/399 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:29.558 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/399 2023-12-17T00:11:29.558 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/399 2023-12-17T00:11:29.558 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:29.561 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:29.568 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:29.569 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/400) 2023-12-17T00:11:29.720 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:29.726 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:29.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:29.839 INFO:teuthology.orchestra.run.smithi028.stdout:generic/400 [not run] disk quotas not supported by this filesystem type: ceph-fuse 2023-12-17T00:11:29.842 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/400 2023-12-17T00:11:29.842 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/400 2023-12-17T00:11:29.842 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:29.844 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:29.852 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:29.853 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/401) 2023-12-17T00:11:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:30.107 INFO:teuthology.orchestra.run.smithi028.stdout:generic/401 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:30.110 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/401 2023-12-17T00:11:30.110 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/401 2023-12-17T00:11:30.111 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:30.121 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:30.121 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/402) 2023-12-17T00:11:30.273 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:30.277 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:30.277 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:30.396 INFO:teuthology.orchestra.run.smithi028.stdout:generic/402 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:30.400 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/402 2023-12-17T00:11:30.400 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/402 2023-12-17T00:11:30.400 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:30.402 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:30.411 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:30.411 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/403) 2023-12-17T00:11:30.560 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:30.564 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:30.564 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:30.667 INFO:teuthology.orchestra.run.smithi028.stdout:generic/403 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:30.671 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/403 2023-12-17T00:11:30.671 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/403 2023-12-17T00:11:30.672 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:30.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:30.681 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:30.682 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/404) 2023-12-17T00:11:30.831 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:30.835 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:30.836 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:32.092 INFO:teuthology.orchestra.run.smithi028.stdout:generic/404 [not run] xfs_io finsert failed (old kernel/wrong fs?) 2023-12-17T00:11:32.097 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/404 2023-12-17T00:11:32.098 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/404 2023-12-17T00:11:32.098 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:32.114 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:32.114 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/405) 2023-12-17T00:11:32.668 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:32.672 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:32.672 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:32.874 INFO:teuthology.orchestra.run.smithi028.stdout:generic/405 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:32.878 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/405 2023-12-17T00:11:32.878 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/405 2023-12-17T00:11:32.878 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:32.882 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:32.891 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:32.891 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/406) 2023-12-17T00:11:33.055 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:33.060 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:33.060 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:33.163 INFO:teuthology.orchestra.run.smithi028.stdout:generic/406 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:33.167 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/406 2023-12-17T00:11:33.167 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/406 2023-12-17T00:11:33.167 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:33.169 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:33.177 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:33.177 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/407) 2023-12-17T00:11:33.328 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:33.331 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:33.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:33.499 INFO:teuthology.orchestra.run.smithi028.stdout:generic/407 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-17T00:11:33.502 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/407 2023-12-17T00:11:33.503 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/407 2023-12-17T00:11:33.503 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:33.505 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:33.513 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:33.513 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/408) 2023-12-17T00:11:34.011 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:34.015 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:34.173 INFO:teuthology.orchestra.run.smithi028.stdout:generic/408 [not run] Dedupe not supported by test filesystem type: ceph-fuse 2023-12-17T00:11:34.176 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/408 2023-12-17T00:11:34.176 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/408 2023-12-17T00:11:34.176 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:34.178 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:34.187 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:34.187 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/409) 2023-12-17T00:11:34.692 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:34.695 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:34.696 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:34.801 INFO:teuthology.orchestra.run.smithi028.stdout:generic/409 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:34.805 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/409 2023-12-17T00:11:34.805 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/409 2023-12-17T00:11:34.805 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:34.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:34.815 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:34.815 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/410) 2023-12-17T00:11:34.963 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:34.967 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:34.967 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:35.071 INFO:teuthology.orchestra.run.smithi028.stdout:generic/410 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:35.075 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/410 2023-12-17T00:11:35.075 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/410 2023-12-17T00:11:35.075 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:35.077 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:35.085 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:35.085 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/411) 2023-12-17T00:11:35.237 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:35.241 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:35.241 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:35.347 INFO:teuthology.orchestra.run.smithi028.stdout:generic/411 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:35.351 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/411 2023-12-17T00:11:35.351 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/411 2023-12-17T00:11:35.351 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:35.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:35.361 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:35.361 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/412) 2023-12-17T00:11:35.513 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:35.517 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:35.618 INFO:teuthology.orchestra.run.smithi028.stdout:generic/412 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/412 2023-12-17T00:11:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/412 2023-12-17T00:11:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:35.624 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:35.632 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:35.632 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/413) 2023-12-17T00:11:35.782 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:35.786 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:35.786 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:35.891 INFO:teuthology.orchestra.run.smithi028.stdout:generic/413 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:35.895 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/413 2023-12-17T00:11:35.895 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/413 2023-12-17T00:11:35.895 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:35.897 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:35.910 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:35.910 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/414) 2023-12-17T00:11:36.060 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:36.064 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:36.064 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:36.166 INFO:teuthology.orchestra.run.smithi028.stdout:generic/414 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:36.170 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/414 2023-12-17T00:11:36.170 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/414 2023-12-17T00:11:36.170 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:36.172 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:36.180 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:36.180 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/415) 2023-12-17T00:11:36.332 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:36.336 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:36.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:36.437 INFO:teuthology.orchestra.run.smithi028.stdout:generic/415 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:36.441 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/415 2023-12-17T00:11:36.441 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/415 2023-12-17T00:11:36.441 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:36.443 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:36.451 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:36.451 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/416) 2023-12-17T00:11:36.602 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:36.606 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:36.607 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:36.710 INFO:teuthology.orchestra.run.smithi028.stdout:generic/416 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:36.713 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/416 2023-12-17T00:11:36.713 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/416 2023-12-17T00:11:36.713 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:36.716 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:36.723 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:36.724 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/417) 2023-12-17T00:11:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:36.876 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:36.978 INFO:teuthology.orchestra.run.smithi028.stdout:generic/417 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:36.981 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/417 2023-12-17T00:11:36.981 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/417 2023-12-17T00:11:36.981 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:36.991 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:36.991 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/418) 2023-12-17T00:11:37.144 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:37.148 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:37.148 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:37.302 INFO:teuthology.orchestra.run.smithi028.stdout:generic/418 [not run] require ceph-fuse to be valid block disk 2023-12-17T00:11:37.306 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/418 2023-12-17T00:11:37.306 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/418 2023-12-17T00:11:37.306 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:37.308 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:37.316 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:37.316 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/419) 2023-12-17T00:11:37.469 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:37.473 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:37.473 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:37.604 INFO:teuthology.orchestra.run.smithi028.stdout:generic/419 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:37.608 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/419 2023-12-17T00:11:37.608 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/419 2023-12-17T00:11:37.608 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:37.610 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:37.618 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:37.618 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/420) 2023-12-17T00:11:37.774 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:37.778 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:37.778 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:38.725 INFO:teuthology.orchestra.run.smithi028.stdout:generic/420 1s 2023-12-17T00:11:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/420 2023-12-17T00:11:38.731 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:38.733 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:38.742 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:38.742 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/421) 2023-12-17T00:11:39.262 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:39.266 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:39.266 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:39.398 INFO:teuthology.orchestra.run.smithi028.stdout:generic/421 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:39.402 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/421 2023-12-17T00:11:39.402 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/421 2023-12-17T00:11:39.402 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:39.404 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:39.412 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:39.412 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/422) 2023-12-17T00:11:39.568 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:39.572 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:39.572 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:39.678 INFO:teuthology.orchestra.run.smithi028.stdout:generic/422 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:39.682 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/422 2023-12-17T00:11:39.682 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/422 2023-12-17T00:11:39.682 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:39.684 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:39.692 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:39.692 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/423) 2023-12-17T00:11:39.844 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:39.848 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:40.099 INFO:teuthology.orchestra.run.smithi028.stdout:generic/423 1s 2023-12-17T00:11:40.104 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/423 2023-12-17T00:11:40.104 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:40.106 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:40.114 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:40.114 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/424) 2023-12-17T00:11:40.597 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:40.601 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:40.602 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:generic/424 [not run] file system doesn't support any of /usr/bin/chattr +a/+c/+d/+i 2023-12-17T00:11:40.891 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/424 2023-12-17T00:11:40.891 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/424 2023-12-17T00:11:40.891 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:40.894 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:40.906 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:40.906 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/425) 2023-12-17T00:11:41.099 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:41.103 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:41.103 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:41.221 INFO:teuthology.orchestra.run.smithi028.stdout:generic/425 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:11:41.225 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/425 2023-12-17T00:11:41.225 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/425 2023-12-17T00:11:41.225 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:11:41.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:11:41.236 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:11:41.236 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/426) 2023-12-17T00:11:41.391 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:11:41.395 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:11:41.395 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:35.151 INFO:teuthology.orchestra.run.smithi028.stdout:generic/426 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/426.out.bad) 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/426.out 2023-12-16 21:58:54.277019508 +0000 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/426.out.bad 2023-12-17 00:12:35.123706531 +0000 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,5 +1,362 @@ 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 426 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: test_file_handles TEST_DIR/426-dir -d 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000206) opened an unlinked file! 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000207) opened an unlinked file! 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000208) opened an unlinked file! 2023-12-17T00:12:35.165 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000209) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000210) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000211) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000212) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000213) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000214) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000215) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000216) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000217) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000218) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000219) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000220) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000221) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000222) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000223) opened an unlinked file! 2023-12-17T00:12:35.166 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000224) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000225) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000226) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000227) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000228) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000229) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000230) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000231) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000232) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000233) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000234) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000235) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000236) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000237) opened an unlinked file! 2023-12-17T00:12:35.167 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000238) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000239) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000240) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000241) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000242) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000243) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000244) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000245) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000246) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000247) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000248) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000249) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000250) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000251) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000252) opened an unlinked file! 2023-12-17T00:12:35.168 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000253) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000254) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000255) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000256) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000257) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000258) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000259) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000260) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000261) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000262) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000263) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000264) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000265) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000266) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000267) opened an unlinked file! 2023-12-17T00:12:35.169 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000268) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000269) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000270) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000271) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000272) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000273) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000274) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000275) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000276) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000277) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000278) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000279) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000280) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000281) opened an unlinked file! 2023-12-17T00:12:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000282) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000283) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000284) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000285) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000286) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000287) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000288) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000289) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000290) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000291) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000292) opened an unlinked file! 2023-12-17T00:12:35.171 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000293) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000294) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000295) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000296) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000297) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000298) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000299) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000300) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000301) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000302) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000303) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000304) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000305) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000306) opened an unlinked file! 2023-12-17T00:12:35.172 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000307) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000308) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000309) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000310) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000311) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000312) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000313) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000314) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000315) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000316) opened an unlinked file! 2023-12-17T00:12:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000317) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000318) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000319) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000320) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000321) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000322) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000784) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000785) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000786) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000787) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000788) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000789) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000790) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000791) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000792) opened an unlinked file! 2023-12-17T00:12:35.174 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000793) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000794) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000795) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000796) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000797) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000798) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000799) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000800) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000801) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000802) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000803) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000804) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000805) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000806) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000807) opened an unlinked file! 2023-12-17T00:12:35.175 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000808) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000809) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000810) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000811) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000812) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000813) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000814) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000815) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000816) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000817) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000818) opened an unlinked file! 2023-12-17T00:12:35.176 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000819) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000820) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000821) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000822) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000823) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000824) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000825) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000826) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000827) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000828) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000829) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000830) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000831) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000832) opened an unlinked file! 2023-12-17T00:12:35.177 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000833) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000834) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000835) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000836) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000837) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000838) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000839) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000840) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000841) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000842) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000843) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000844) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000845) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000846) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000847) opened an unlinked file! 2023-12-17T00:12:35.178 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000848) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000849) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000850) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000851) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000852) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000853) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000854) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000855) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000856) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000857) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000858) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000859) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000860) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000861) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000862) opened an unlinked file! 2023-12-17T00:12:35.179 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000863) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000864) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000865) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000866) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000867) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000868) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000869) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000870) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000871) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000872) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000873) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000874) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000875) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000876) opened an unlinked file! 2023-12-17T00:12:35.180 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000877) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000878) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000879) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000880) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000881) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000882) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000883) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000884) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000885) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000886) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000887) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000888) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000889) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000890) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000891) opened an unlinked file! 2023-12-17T00:12:35.181 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000892) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000893) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000894) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000895) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000896) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000897) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000898) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000899) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000900) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000901) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000902) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000903) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000904) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000905) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000906) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000907) opened an unlinked file! 2023-12-17T00:12:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000908) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000909) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000910) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000911) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000912) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000913) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000914) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000915) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000916) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000917) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000918) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000919) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000920) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000921) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000922) opened an unlinked file! 2023-12-17T00:12:35.183 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000923) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000924) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000925) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000926) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000927) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000928) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000929) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000930) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000931) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000932) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000933) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000934) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000935) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000936) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000937) opened an unlinked file! 2023-12-17T00:12:35.184 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000938) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000939) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000940) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000941) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000942) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000943) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000944) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000945) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000946) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000947) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000948) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000949) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000950) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000951) opened an unlinked file! 2023-12-17T00:12:35.185 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000952) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000953) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000954) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000955) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000956) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000957) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000958) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000959) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000960) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000961) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000962) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000963) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000964) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000965) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000966) opened an unlinked file! 2023-12-17T00:12:35.186 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000967) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000968) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000969) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000970) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000971) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000972) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000973) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000974) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000975) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000976) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000977) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000978) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000979) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000980) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000981) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000982) opened an unlinked file! 2023-12-17T00:12:35.187 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000983) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000984) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000985) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000986) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000987) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000988) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000989) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000990) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000991) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000992) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000993) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000994) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000995) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000996) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000997) opened an unlinked file! 2023-12-17T00:12:35.188 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000998) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file000999) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001000) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001001) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001002) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001003) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001004) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001005) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001006) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001007) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001008) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001009) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001010) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001011) opened an unlinked file! 2023-12-17T00:12:35.189 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001012) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001013) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001014) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001015) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001016) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001017) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001018) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001019) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001020) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001021) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001022) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/426-dir/file001023) opened an unlinked file! 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: test_file_handles TEST_DIR/426-dir 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: test_file_handles TEST_DIR/426-dir -l 2023-12-17T00:12:35.190 INFO:teuthology.orchestra.run.smithi028.stdout: test_file_handles TEST_DIR/426-dir -u 2023-12-17T00:12:35.191 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/426 2023-12-17T00:12:35.191 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/426 2023-12-17T00:12:35.191 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T00:12:35.218 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T00:12:35.285 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T00:12:35.285 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T00:12:35.285 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/427) 2023-12-17T00:12:36.961 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:36.967 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:36.967 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:37.175 INFO:teuthology.orchestra.run.smithi028.stdout:generic/427 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:12:37.180 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/427 2023-12-17T00:12:37.181 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/427 2023-12-17T00:12:37.181 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:37.184 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:37.194 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:37.194 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/428) 2023-12-17T00:12:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:37.395 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:37.395 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:37.582 INFO:teuthology.orchestra.run.smithi028.stdout:generic/428 0s 2023-12-17T00:12:37.589 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/428 2023-12-17T00:12:37.589 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:37.592 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:37.601 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:37.602 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/429) 2023-12-17T00:12:37.780 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:37.785 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:37.785 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:37.945 INFO:teuthology.orchestra.run.smithi028.stdout:generic/429 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:12:37.948 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/429 2023-12-17T00:12:37.948 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/429 2023-12-17T00:12:37.948 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:37.951 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:37.960 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:37.960 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/430) 2023-12-17T00:12:38.119 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:38.122 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:38.123 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:38.384 INFO:teuthology.orchestra.run.smithi028.stdout:generic/430 0s 2023-12-17T00:12:38.389 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/430 2023-12-17T00:12:38.389 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:38.391 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:38.399 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:38.399 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/431) 2023-12-17T00:12:38.897 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:38.900 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:38.901 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:39.135 INFO:teuthology.orchestra.run.smithi028.stdout:generic/431 1s 2023-12-17T00:12:39.140 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/431 2023-12-17T00:12:39.140 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:39.142 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:39.150 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:39.150 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/432) 2023-12-17T00:12:39.652 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:39.656 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:39.656 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:39.918 INFO:teuthology.orchestra.run.smithi028.stdout:generic/432 0s 2023-12-17T00:12:39.923 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/432 2023-12-17T00:12:39.923 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:39.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:39.933 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:39.934 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/433) 2023-12-17T00:12:40.452 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:40.456 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:40.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:40.667 INFO:teuthology.orchestra.run.smithi028.stdout:generic/433 0s 2023-12-17T00:12:40.672 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/433 2023-12-17T00:12:40.672 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:40.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:40.683 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:40.683 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/434) 2023-12-17T00:12:41.195 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:41.199 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:41.199 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:41.426 INFO:teuthology.orchestra.run.smithi028.stdout:generic/434 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/434.out.bad) 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/434.out 2023-12-16 21:58:54.277019508 +0000 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/434.out.bad 2023-12-17 00:12:41.427499190 +0000 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -9,6 +9,6 @@ 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: copy_range: Bad file descriptor 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: d41d8cd98f00b204e9800998ecf8427e TEST_DIR/test-434/copy 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: Try to copy to a device 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: -copy_range: Invalid argument 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: +/tmp/tmp.XR9uc7m4Witest/test-434/dev1: Permission denied 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: Try to copy to a fifo 2023-12-17T00:12:41.428 INFO:teuthology.orchestra.run.smithi028.stdout: copy_range: Invalid argument 2023-12-17T00:12:41.430 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/434 2023-12-17T00:12:41.430 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/434 2023-12-17T00:12:41.430 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T00:12:41.432 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T00:12:41.441 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T00:12:41.441 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T00:12:41.441 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/435) 2023-12-17T00:12:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:41.935 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:42.066 INFO:teuthology.orchestra.run.smithi028.stdout:generic/435 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:12:42.069 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/435 2023-12-17T00:12:42.069 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/435 2023-12-17T00:12:42.069 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:42.080 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:42.080 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/436) 2023-12-17T00:12:42.228 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:42.232 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:44.173 INFO:teuthology.orchestra.run.smithi028.stdout:generic/436 2s 2023-12-17T00:12:44.181 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/436 2023-12-17T00:12:44.181 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:44.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:44.462 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:44.462 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/437) 2023-12-17T00:12:44.777 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:44.781 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:44.781 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:45.670 INFO:teuthology.orchestra.run.smithi028.stdout:generic/437 1s 2023-12-17T00:12:45.677 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/437 2023-12-17T00:12:45.677 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:12:45.681 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:12:45.732 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:12:45.733 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/438) 2023-12-17T00:12:45.944 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:12:45.948 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:12:45.949 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:54.011 INFO:teuthology.orchestra.run.smithi028.stdout:generic/438 68s 2023-12-17T00:13:54.020 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/438 2023-12-17T00:13:54.020 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:54.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:54.036 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:54.036 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/439) 2023-12-17T00:13:54.202 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:54.206 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:54.206 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:54.310 INFO:teuthology.orchestra.run.smithi028.stdout:generic/439 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:13:54.314 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/439 2023-12-17T00:13:54.314 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/439 2023-12-17T00:13:54.314 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:54.316 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:54.324 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:54.324 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/440) 2023-12-17T00:13:54.476 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:54.480 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:54.480 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:54.613 INFO:teuthology.orchestra.run.smithi028.stdout:generic/440 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:13:54.617 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/440 2023-12-17T00:13:54.617 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/440 2023-12-17T00:13:54.617 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:54.628 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:54.629 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/441) 2023-12-17T00:13:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:54.802 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:54.802 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:54.996 INFO:teuthology.orchestra.run.smithi028.stdout:generic/441 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:13:55.001 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/441 2023-12-17T00:13:55.001 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/441 2023-12-17T00:13:55.001 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:55.004 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:55.012 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:55.013 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/442) 2023-12-17T00:13:55.175 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:55.179 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:55.179 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:55.298 INFO:teuthology.orchestra.run.smithi028.stdout:generic/442 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:13:55.301 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/442 2023-12-17T00:13:55.301 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/442 2023-12-17T00:13:55.302 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:55.304 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:55.312 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:55.312 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/443) 2023-12-17T00:13:55.462 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:55.466 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:55.466 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:55.612 INFO:teuthology.orchestra.run.smithi028.stdout:generic/443 0s 2023-12-17T00:13:55.617 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/443 2023-12-17T00:13:55.617 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:55.627 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:55.627 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/444) 2023-12-17T00:13:55.780 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:55.784 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:55.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:55.938 INFO:teuthology.orchestra.run.smithi028.stdout:generic/444 [not run] ACLs not supported by this filesystem type: ceph-fuse 2023-12-17T00:13:55.942 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/444 2023-12-17T00:13:55.942 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/444 2023-12-17T00:13:55.942 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:55.945 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:55.952 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:55.952 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/445) 2023-12-17T00:13:56.105 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:56.109 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:56.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:57.853 INFO:teuthology.orchestra.run.smithi028.stdout:generic/445 1s 2023-12-17T00:13:57.861 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/445 2023-12-17T00:13:57.861 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:57.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:57.876 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:57.877 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/446) 2023-12-17T00:13:58.442 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:58.446 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:58.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:58.549 INFO:teuthology.orchestra.run.smithi028.stdout:generic/446 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:13:58.553 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/446 2023-12-17T00:13:58.553 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/446 2023-12-17T00:13:58.553 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:58.555 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:58.563 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:58.563 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/447) 2023-12-17T00:13:58.717 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:58.721 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:58.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:58.824 INFO:teuthology.orchestra.run.smithi028.stdout:generic/447 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:13:58.828 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/447 2023-12-17T00:13:58.828 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/447 2023-12-17T00:13:58.828 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:13:58.830 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:13:58.838 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:13:58.838 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/448) 2023-12-17T00:13:58.993 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:13:58.997 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:13:58.997 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:00.747 INFO:teuthology.orchestra.run.smithi028.stdout:generic/448 1s 2023-12-17T00:14:00.755 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/448 2023-12-17T00:14:00.755 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:00.758 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:00.769 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:00.770 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/449) 2023-12-17T00:14:01.327 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:01.331 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:01.331 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:01.442 INFO:teuthology.orchestra.run.smithi028.stdout:generic/449 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:01.445 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/449 2023-12-17T00:14:01.445 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/449 2023-12-17T00:14:01.446 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:01.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:01.456 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:01.456 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/450) 2023-12-17T00:14:01.612 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:01.616 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:01.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:02.988 INFO:teuthology.orchestra.run.smithi028.stdout:generic/450 1s 2023-12-17T00:14:02.996 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/450 2023-12-17T00:14:02.996 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:02.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:03.012 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:03.013 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/451) 2023-12-17T00:14:03.567 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:03.571 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:04.021 INFO:teuthology.orchestra.run.smithi028.stdout:generic/451 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/451.out.bad) 2023-12-17T00:14:04.022 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/451.out 2023-12-16 21:58:54.277019508 +0000 2023-12-17T00:14:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/451.out.bad 2023-12-17 00:14:04.020760751 +0000 2023-12-17T00:14:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,2 +1,5 @@ 2023-12-17T00:14:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 451 2023-12-17T00:14:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: +get stale data from buffer read 2023-12-17T00:14:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: +00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 2023-12-17T00:14:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: +000a0000 2023-12-17T00:14:04.023 INFO:teuthology.orchestra.run.smithi028.stdout: Silence is golden 2023-12-17T00:14:04.025 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/451 2023-12-17T00:14:04.025 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/451 2023-12-17T00:14:04.025 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T00:14:04.027 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T00:14:04.036 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T00:14:04.036 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T00:14:04.036 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/452) 2023-12-17T00:14:04.499 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:04.514 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:04.514 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:04.651 INFO:teuthology.orchestra.run.smithi028.stdout:generic/452 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:04.655 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/452 2023-12-17T00:14:04.655 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/452 2023-12-17T00:14:04.655 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:04.657 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:04.665 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:04.665 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/453) 2023-12-17T00:14:04.814 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:04.817 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:04.817 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:04.916 INFO:teuthology.orchestra.run.smithi028.stdout:generic/453 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:04.920 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/453 2023-12-17T00:14:04.920 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/453 2023-12-17T00:14:04.920 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:04.930 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:04.930 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/454) 2023-12-17T00:14:05.083 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:05.087 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:05.190 INFO:teuthology.orchestra.run.smithi028.stdout:generic/454 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:05.193 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/454 2023-12-17T00:14:05.193 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/454 2023-12-17T00:14:05.193 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:05.203 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:05.203 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/455) 2023-12-17T00:14:05.354 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:05.358 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:05.554 INFO:teuthology.orchestra.run.smithi028.stdout:generic/455 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:05.558 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/455 2023-12-17T00:14:05.558 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/455 2023-12-17T00:14:05.558 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:05.569 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:05.570 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/456) 2023-12-17T00:14:05.728 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:05.732 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:05.854 INFO:teuthology.orchestra.run.smithi028.stdout:generic/456 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:05.858 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/456 2023-12-17T00:14:05.858 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/456 2023-12-17T00:14:05.858 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:05.868 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:05.868 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/457) 2023-12-17T00:14:06.018 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:06.021 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:06.246 INFO:teuthology.orchestra.run.smithi028.stdout:generic/457 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:06.251 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/457 2023-12-17T00:14:06.251 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/457 2023-12-17T00:14:06.251 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:06.261 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:06.261 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/458) 2023-12-17T00:14:06.410 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:06.414 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:06.515 INFO:teuthology.orchestra.run.smithi028.stdout:generic/458 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/458 2023-12-17T00:14:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/458 2023-12-17T00:14:06.519 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:06.529 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:06.529 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/459) 2023-12-17T00:14:06.679 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:06.683 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:06.934 INFO:teuthology.orchestra.run.smithi028.stdout:generic/459 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:06.940 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/459 2023-12-17T00:14:06.940 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/459 2023-12-17T00:14:06.940 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:06.955 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:06.955 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/460) 2023-12-17T00:14:07.145 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:07.150 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:07.253 INFO:teuthology.orchestra.run.smithi028.stdout:generic/460 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:07.257 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/460 2023-12-17T00:14:07.257 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/460 2023-12-17T00:14:07.257 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:07.267 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:07.267 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/461) 2023-12-17T00:14:07.418 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:07.422 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:07.423 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:07.525 INFO:teuthology.orchestra.run.smithi028.stdout:generic/461 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/461 2023-12-17T00:14:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/461 2023-12-17T00:14:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:07.532 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:07.540 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:07.540 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/462) 2023-12-17T00:14:07.694 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:07.698 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:07.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:07.810 INFO:teuthology.orchestra.run.smithi028.stdout:generic/462 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:07.814 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/462 2023-12-17T00:14:07.814 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/462 2023-12-17T00:14:07.814 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:07.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:07.825 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:07.825 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/463) 2023-12-17T00:14:07.978 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:07.982 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:07.982 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:08.145 INFO:teuthology.orchestra.run.smithi028.stdout:generic/463 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-17T00:14:08.149 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/463 2023-12-17T00:14:08.149 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/463 2023-12-17T00:14:08.149 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:08.159 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:08.159 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/464) 2023-12-17T00:14:08.636 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:08.640 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:08.640 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:08.741 INFO:teuthology.orchestra.run.smithi028.stdout:generic/464 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:08.744 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/464 2023-12-17T00:14:08.744 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/464 2023-12-17T00:14:08.745 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:08.747 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:08.754 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:08.754 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/465) 2023-12-17T00:14:08.904 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:08.908 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:08.908 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:32.532 INFO:teuthology.orchestra.run.smithi028.stdout:generic/465 24s 2023-12-17T00:14:32.612 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/465 2023-12-17T00:14:32.612 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:32.691 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:32.735 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:32.736 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/466) 2023-12-17T00:14:32.920 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:32.924 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:32.924 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:33.030 INFO:teuthology.orchestra.run.smithi028.stdout:generic/466 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:33.034 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/466 2023-12-17T00:14:33.034 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/466 2023-12-17T00:14:33.034 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:33.045 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:33.045 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/467) 2023-12-17T00:14:33.227 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:33.230 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:33.231 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:40.028 INFO:teuthology.orchestra.run.smithi028.stdout:generic/467 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/467.out.bad) 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/467.out 2023-12-16 21:58:54.281019341 +0000 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/467.out.bad 2023-12-17 00:14:40.007548579 +0000 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,5 +1,15 @@ 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 467 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: test_file_handles TEST_DIR/467-dir -dp 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000000) opened an unlinked file! 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000001) opened an unlinked file! 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000002) opened an unlinked file! 2023-12-17T00:14:40.041 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000003) opened an unlinked file! 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000004) opened an unlinked file! 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000005) opened an unlinked file! 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000006) opened an unlinked file! 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000007) opened an unlinked file! 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000008) opened an unlinked file! 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: +open_by_handle(/tmp/tmp.XR9uc7m4Witest/467-dir/file000009) opened an unlinked file! 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: test_file_handles TEST_DIR/467-dir -rp 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: test_file_handles TEST_DIR/467-dir -dkr 2023-12-17T00:14:40.042 INFO:teuthology.orchestra.run.smithi028.stdout: test_file_handles TEST_DIR/467-dir -lr 2023-12-17T00:14:40.062 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/467 2023-12-17T00:14:40.062 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/467 2023-12-17T00:14:40.062 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T00:14:40.091 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T00:14:40.161 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T00:14:40.161 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T00:14:40.161 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/468) 2023-12-17T00:14:41.784 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:41.790 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:41.790 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:41.980 INFO:teuthology.orchestra.run.smithi028.stdout:generic/468 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/468 2023-12-17T00:14:41.985 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/468 2023-12-17T00:14:41.986 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:41.999 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:41.999 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/469) 2023-12-17T00:14:42.198 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:42.203 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:45.199 INFO:teuthology.orchestra.run.smithi028.stdout:generic/469 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-12-17T00:14:45.204 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/469 2023-12-17T00:14:45.204 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/469 2023-12-17T00:14:45.204 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:45.210 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:45.217 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:45.218 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/470) 2023-12-17T00:14:45.771 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:45.776 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:45.776 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:45.964 INFO:teuthology.orchestra.run.smithi028.stdout:generic/470 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:14:45.969 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/470 2023-12-17T00:14:45.969 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/470 2023-12-17T00:14:45.970 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:14:45.974 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:14:45.984 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:14:45.985 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/471) 2023-12-17T00:14:46.207 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:14:46.211 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:14:46.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:05.678 INFO:teuthology.orchestra.run.smithi028.stdout:generic/471 79s 2023-12-17T00:16:05.684 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/471 2023-12-17T00:16:05.684 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:05.741 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:05.741 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/472) 2023-12-17T00:16:06.776 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:06.787 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:07.141 INFO:teuthology.orchestra.run.smithi028.stdout:generic/472 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:07.146 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/472 2023-12-17T00:16:07.146 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/472 2023-12-17T00:16:07.146 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:07.185 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:07.185 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/473) 2023-12-17T00:16:07.599 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:07.603 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:07.769 INFO:teuthology.orchestra.run.smithi028.stdout:generic/473 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-17T00:16:07.772 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/473 2023-12-17T00:16:07.772 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/473 2023-12-17T00:16:07.773 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:07.775 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:07.783 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:07.783 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/474) 2023-12-17T00:16:07.943 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:07.947 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:07.948 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:08.050 INFO:teuthology.orchestra.run.smithi028.stdout:generic/474 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:08.054 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/474 2023-12-17T00:16:08.054 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/474 2023-12-17T00:16:08.054 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:08.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:08.064 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:08.065 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/475) 2023-12-17T00:16:08.230 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:08.234 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:08.234 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:08.387 INFO:teuthology.orchestra.run.smithi028.stdout:generic/475 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:08.392 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/475 2023-12-17T00:16:08.393 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/475 2023-12-17T00:16:08.393 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:08.393 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:08.402 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:08.402 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/476) 2023-12-17T00:16:08.552 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:08.556 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:08.556 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:08.660 INFO:teuthology.orchestra.run.smithi028.stdout:generic/476 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:08.664 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/476 2023-12-17T00:16:08.664 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/476 2023-12-17T00:16:08.664 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:08.666 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:08.675 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:08.675 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/477) 2023-12-17T00:16:08.821 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:08.825 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:08.825 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:18.942 INFO:teuthology.orchestra.run.smithi028.stdout:generic/477 10s 2023-12-17T00:16:19.108 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/477 2023-12-17T00:16:19.108 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:19.114 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:19.189 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:19.189 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/478) 2023-12-17T00:16:20.638 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:20.644 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:20.645 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:31.522 INFO:teuthology.orchestra.run.smithi028.stdout:generic/478 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/478.out.bad) 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/478.out 2023-12-16 21:58:54.281019341 +0000 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/478.out.bad 2023-12-17 00:16:31.519839250 +0000 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -3,14 +3,14 @@ 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: get wrlck 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: get wrlck 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: -lock could be placed 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: +get wrlck 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: get wrlck 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.525 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: get wrlck 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: -lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: +get wrlck 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -25,7 +25,7 @@ 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: get wrlck 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: -lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: +get wrlck 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.526 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -37,7 +37,7 @@ 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: get rdlck 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: get rdlck 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: -lock could be placed 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: +get rdlck 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: get rdlck 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: get rdlck 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -46,7 +46,7 @@ 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: get rdlck 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: get rdlck 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.527 INFO:teuthology.orchestra.run.smithi028.stdout: -lock could be placed 2023-12-17T00:16:31.528 INFO:teuthology.orchestra.run.smithi028.stdout: +get rdlck 2023-12-17T00:16:31.528 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.528 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.528 INFO:teuthology.orchestra.run.smithi028.stdout: lock could be placed 2023-12-17T00:16:31.528 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/478 2023-12-17T00:16:31.528 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/478 2023-12-17T00:16:31.529 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T00:16:31.533 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T00:16:31.547 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T00:16:31.547 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T00:16:31.547 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/479) 2023-12-17T00:16:31.748 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:31.752 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:31.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:31.922 INFO:teuthology.orchestra.run.smithi028.stdout:generic/479 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:31.926 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/479 2023-12-17T00:16:31.926 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/479 2023-12-17T00:16:31.926 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:31.929 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:31.939 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:31.939 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/480) 2023-12-17T00:16:32.114 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:32.118 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:32.118 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:32.273 INFO:teuthology.orchestra.run.smithi028.stdout:generic/480 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:32.278 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/480 2023-12-17T00:16:32.278 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/480 2023-12-17T00:16:32.278 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:32.280 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:32.290 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:32.290 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/481) 2023-12-17T00:16:32.470 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:32.474 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:32.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:32.604 INFO:teuthology.orchestra.run.smithi028.stdout:generic/481 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:32.607 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/481 2023-12-17T00:16:32.607 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/481 2023-12-17T00:16:32.608 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:32.610 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:32.618 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:32.618 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/482) 2023-12-17T00:16:32.776 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:32.780 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:32.780 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:32.972 INFO:teuthology.orchestra.run.smithi028.stdout:generic/482 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:32.976 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/482 2023-12-17T00:16:32.976 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/482 2023-12-17T00:16:32.976 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:32.979 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:32.987 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:32.987 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/483) 2023-12-17T00:16:33.143 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:33.147 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:33.147 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:33.276 INFO:teuthology.orchestra.run.smithi028.stdout:generic/483 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:33.279 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/483 2023-12-17T00:16:33.279 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/483 2023-12-17T00:16:33.279 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:33.282 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:33.290 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:33.290 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/484) 2023-12-17T00:16:33.455 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:33.459 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:33.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:33.574 INFO:teuthology.orchestra.run.smithi028.stdout:generic/484 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:33.578 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/484 2023-12-17T00:16:33.578 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/484 2023-12-17T00:16:33.578 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:33.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:33.589 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:33.590 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/485) 2023-12-17T00:16:33.738 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:33.742 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:33.742 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:35.486 INFO:teuthology.orchestra.run.smithi028.stdout:generic/485 [not run] xfs_io finsert failed (old kernel/wrong fs?) 2023-12-17T00:16:35.493 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/485 2023-12-17T00:16:35.493 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/485 2023-12-17T00:16:35.493 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:35.497 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:35.509 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:35.509 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/486) 2023-12-17T00:16:36.084 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:36.088 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:36.088 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:36.232 INFO:teuthology.orchestra.run.smithi028.stdout:generic/486 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:36.235 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/486 2023-12-17T00:16:36.236 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/486 2023-12-17T00:16:36.236 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:36.238 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:36.246 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:36.247 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/487) 2023-12-17T00:16:36.396 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:36.400 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:36.401 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:36.534 INFO:teuthology.orchestra.run.smithi028.stdout:generic/487 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:36.538 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/487 2023-12-17T00:16:36.538 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/487 2023-12-17T00:16:36.538 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:36.540 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:36.549 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:36.549 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/488) 2023-12-17T00:16:36.697 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:36.702 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:36.702 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:36.803 INFO:teuthology.orchestra.run.smithi028.stdout:generic/488 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:36.807 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/488 2023-12-17T00:16:36.807 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/488 2023-12-17T00:16:36.807 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:36.817 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:36.818 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/489) 2023-12-17T00:16:36.964 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:36.968 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:36.968 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:37.103 INFO:teuthology.orchestra.run.smithi028.stdout:generic/489 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:37.107 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/489 2023-12-17T00:16:37.107 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/489 2023-12-17T00:16:37.107 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:37.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:37.117 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:37.117 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/490) 2023-12-17T00:16:37.263 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:37.267 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:37.267 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:38.795 INFO:teuthology.orchestra.run.smithi028.stdout:generic/490 1s 2023-12-17T00:16:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/490 2023-12-17T00:16:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:38.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:39.022 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:39.022 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/491) 2023-12-17T00:16:39.413 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:39.417 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:39.417 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:39.542 INFO:teuthology.orchestra.run.smithi028.stdout:generic/491 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:39.545 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/491 2023-12-17T00:16:39.545 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/491 2023-12-17T00:16:39.545 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:39.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:39.556 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:39.557 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/492) 2023-12-17T00:16:39.706 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:39.711 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:39.711 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:39.813 INFO:teuthology.orchestra.run.smithi028.stdout:generic/492 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:39.816 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/492 2023-12-17T00:16:39.816 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/492 2023-12-17T00:16:39.816 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:39.828 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:39.828 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/493) 2023-12-17T00:16:39.974 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:39.978 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:39.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:40.080 INFO:teuthology.orchestra.run.smithi028.stdout:generic/493 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:40.084 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/493 2023-12-17T00:16:40.084 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/493 2023-12-17T00:16:40.084 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:40.086 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:40.094 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:40.094 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/494) 2023-12-17T00:16:40.240 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:40.243 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:40.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:40.348 INFO:teuthology.orchestra.run.smithi028.stdout:generic/494 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:40.352 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/494 2023-12-17T00:16:40.352 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/494 2023-12-17T00:16:40.352 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:40.354 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:40.363 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:40.363 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/495) 2023-12-17T00:16:40.510 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:40.514 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:40.514 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:40.615 INFO:teuthology.orchestra.run.smithi028.stdout:generic/495 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:40.619 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/495 2023-12-17T00:16:40.619 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/495 2023-12-17T00:16:40.619 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:40.621 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:40.628 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:40.629 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/496) 2023-12-17T00:16:40.777 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:40.781 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:40.781 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:40.886 INFO:teuthology.orchestra.run.smithi028.stdout:generic/496 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:40.890 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/496 2023-12-17T00:16:40.890 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/496 2023-12-17T00:16:40.890 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:40.892 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:40.900 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:40.901 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/497) 2023-12-17T00:16:41.048 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:41.051 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:41.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:41.155 INFO:teuthology.orchestra.run.smithi028.stdout:generic/497 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:41.159 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/497 2023-12-17T00:16:41.159 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/497 2023-12-17T00:16:41.159 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:41.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:41.170 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:41.170 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/498) 2023-12-17T00:16:41.320 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:41.324 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:41.452 INFO:teuthology.orchestra.run.smithi028.stdout:generic/498 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:41.455 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/498 2023-12-17T00:16:41.455 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/498 2023-12-17T00:16:41.455 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:41.458 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:41.466 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:41.466 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/499) 2023-12-17T00:16:41.613 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:41.617 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:41.617 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:41.721 INFO:teuthology.orchestra.run.smithi028.stdout:generic/499 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:41.726 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/499 2023-12-17T00:16:41.726 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/499 2023-12-17T00:16:41.726 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:41.736 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:41.736 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/500) 2023-12-17T00:16:41.893 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:41.897 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:42.076 INFO:teuthology.orchestra.run.smithi028.stdout:generic/500 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:42.080 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/500 2023-12-17T00:16:42.080 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/500 2023-12-17T00:16:42.080 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:42.091 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:42.091 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/501) 2023-12-17T00:16:42.256 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:42.260 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:42.376 INFO:teuthology.orchestra.run.smithi028.stdout:generic/501 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:42.381 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/501 2023-12-17T00:16:42.381 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/501 2023-12-17T00:16:42.382 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:42.389 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:42.390 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/502) 2023-12-17T00:16:42.538 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:42.542 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:42.663 INFO:teuthology.orchestra.run.smithi028.stdout:generic/502 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:42.667 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/502 2023-12-17T00:16:42.667 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/502 2023-12-17T00:16:42.667 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:42.677 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:42.677 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/503) 2023-12-17T00:16:42.823 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:42.827 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:42.932 INFO:teuthology.orchestra.run.smithi028.stdout:generic/503 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:42.936 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/503 2023-12-17T00:16:42.936 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/503 2023-12-17T00:16:42.936 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:42.946 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:42.946 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/504) 2023-12-17T00:16:43.091 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:43.095 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:43.271 INFO:teuthology.orchestra.run.smithi028.stdout:generic/504 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/504.out.bad) 2023-12-17T00:16:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/504.out 2023-12-16 21:58:54.281019341 +0000 2023-12-17T00:16:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/504.out.bad 2023-12-17 00:16:43.271452263 +0000 2023-12-17T00:16:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,2 +1,3 @@ 2023-12-17T00:16:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 504 2023-12-17T00:16:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: +lock info not found 2023-12-17T00:16:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: Silence is golden 2023-12-17T00:16:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/504 2023-12-17T00:16:43.276 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/504 2023-12-17T00:16:43.277 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T00:16:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T00:16:43.287 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T00:16:43.287 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T00:16:43.287 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/505) 2023-12-17T00:16:43.442 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:43.446 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:43.549 INFO:teuthology.orchestra.run.smithi028.stdout:generic/505 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:43.552 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/505 2023-12-17T00:16:43.552 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/505 2023-12-17T00:16:43.552 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:43.563 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:43.563 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/506) 2023-12-17T00:16:43.708 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:43.712 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:43.817 INFO:teuthology.orchestra.run.smithi028.stdout:generic/506 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:43.818 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/506 2023-12-17T00:16:43.818 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/506 2023-12-17T00:16:43.818 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:43.828 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:43.829 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/507) 2023-12-17T00:16:43.975 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:43.979 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:43.979 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:44.193 INFO:teuthology.orchestra.run.smithi028.stdout:generic/507 [not run] file system doesn't support chattr +ASai 2023-12-17T00:16:44.197 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/507 2023-12-17T00:16:44.198 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/507 2023-12-17T00:16:44.198 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:44.200 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:44.209 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:44.209 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/508) 2023-12-17T00:16:44.365 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:44.369 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:44.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:44.474 INFO:teuthology.orchestra.run.smithi028.stdout:generic/508 [not run] lsattr not supported by test filesystem type: ceph-fuse 2023-12-17T00:16:44.477 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/508 2023-12-17T00:16:44.478 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/508 2023-12-17T00:16:44.478 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:44.480 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:44.487 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:44.488 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/509) 2023-12-17T00:16:44.634 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:44.638 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:44.638 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:44.763 INFO:teuthology.orchestra.run.smithi028.stdout:generic/509 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:44.767 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/509 2023-12-17T00:16:44.767 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/509 2023-12-17T00:16:44.767 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:44.769 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:44.777 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:44.777 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/510) 2023-12-17T00:16:44.923 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:44.927 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:44.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:45.043 INFO:teuthology.orchestra.run.smithi028.stdout:generic/510 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:45.047 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/510 2023-12-17T00:16:45.047 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/510 2023-12-17T00:16:45.047 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:45.049 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:45.057 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:45.058 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/511) 2023-12-17T00:16:45.204 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:45.207 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:45.208 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:45.309 INFO:teuthology.orchestra.run.smithi028.stdout:generic/511 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/511 2023-12-17T00:16:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/511 2023-12-17T00:16:45.313 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:45.316 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:45.324 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:45.324 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/512) 2023-12-17T00:16:45.471 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:45.475 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:45.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:45.607 INFO:teuthology.orchestra.run.smithi028.stdout:generic/512 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:45.611 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/512 2023-12-17T00:16:45.611 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/512 2023-12-17T00:16:45.611 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:45.613 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:45.621 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:45.621 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/513) 2023-12-17T00:16:45.766 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:45.770 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:45.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:45.875 INFO:teuthology.orchestra.run.smithi028.stdout:generic/513 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:45.879 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/513 2023-12-17T00:16:45.879 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/513 2023-12-17T00:16:45.879 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:45.881 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:45.889 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:45.890 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/514) 2023-12-17T00:16:46.045 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:46.049 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:46.049 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:46.154 INFO:teuthology.orchestra.run.smithi028.stdout:generic/514 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:46.158 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/514 2023-12-17T00:16:46.158 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/514 2023-12-17T00:16:46.158 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:46.160 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:46.169 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:46.169 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/515) 2023-12-17T00:16:46.314 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:46.318 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:46.319 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:46.430 INFO:teuthology.orchestra.run.smithi028.stdout:generic/515 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:46.433 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/515 2023-12-17T00:16:46.433 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/515 2023-12-17T00:16:46.433 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:46.435 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:46.443 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:46.444 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/516) 2023-12-17T00:16:46.592 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:46.596 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:46.596 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:46.745 INFO:teuthology.orchestra.run.smithi028.stdout:generic/516 [not run] Dedupe not supported by test filesystem type: ceph-fuse 2023-12-17T00:16:46.748 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/516 2023-12-17T00:16:46.748 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/516 2023-12-17T00:16:46.748 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:46.758 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:46.758 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/517) 2023-12-17T00:16:47.258 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:47.262 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:47.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:47.366 INFO:teuthology.orchestra.run.smithi028.stdout:generic/517 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:47.370 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/517 2023-12-17T00:16:47.370 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/517 2023-12-17T00:16:47.370 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:47.373 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:47.381 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:47.382 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/518) 2023-12-17T00:16:47.529 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:47.533 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:47.533 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:47.638 INFO:teuthology.orchestra.run.smithi028.stdout:generic/518 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:47.642 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/518 2023-12-17T00:16:47.642 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/518 2023-12-17T00:16:47.642 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:47.644 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:47.653 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:47.653 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/519) 2023-12-17T00:16:47.802 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:47.805 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:47.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:47.914 INFO:teuthology.orchestra.run.smithi028.stdout:generic/519 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:47.921 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/519 2023-12-17T00:16:47.921 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/519 2023-12-17T00:16:47.921 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:47.921 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:47.929 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:47.929 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/520) 2023-12-17T00:16:48.076 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:48.080 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:48.080 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:48.195 INFO:teuthology.orchestra.run.smithi028.stdout:generic/520 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T00:16:48.199 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/520 2023-12-17T00:16:48.199 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/520 2023-12-17T00:16:48.199 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T00:16:48.202 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T00:16:48.209 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T00:16:48.209 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/521) 2023-12-17T00:16:48.355 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T00:16:48.359 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T00:16:48.359 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T01:58:32.013 INFO:teuthology.orchestra.run.smithi028.stdout:generic/521 6103s 2023-12-17T01:58:32.070 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/521 2023-12-17T01:58:32.070 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T01:58:32.074 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T01:58:32.086 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T01:58:32.087 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/522) 2023-12-17T01:58:33.046 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T01:58:33.051 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T01:58:33.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:06.031 INFO:teuthology.orchestra.run.smithi028.stdout:generic/522 4773s 2023-12-17T03:18:06.056 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/522 2023-12-17T03:18:06.057 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:06.057 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:06.058 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/523) 2023-12-17T03:18:06.274 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:06.279 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:06.423 INFO:teuthology.orchestra.run.smithi028.stdout:generic/523 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:06.428 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/523 2023-12-17T03:18:06.428 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/523 2023-12-17T03:18:06.428 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:06.440 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:06.440 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/524) 2023-12-17T03:18:06.633 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:06.638 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:06.824 INFO:teuthology.orchestra.run.smithi028.stdout:generic/524 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:06.829 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/524 2023-12-17T03:18:06.829 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/524 2023-12-17T03:18:06.830 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:06.844 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:06.845 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/525) 2023-12-17T03:18:07.043 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:07.048 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:07.175 INFO:teuthology.orchestra.run.smithi028.stdout:generic/525 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:07.179 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/525 2023-12-17T03:18:07.179 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/525 2023-12-17T03:18:07.179 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:07.182 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:07.192 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:07.192 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/526) 2023-12-17T03:18:07.379 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:07.383 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:07.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:07.562 INFO:teuthology.orchestra.run.smithi028.stdout:generic/526 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:07.566 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/526 2023-12-17T03:18:07.567 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/526 2023-12-17T03:18:07.567 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:07.570 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:07.579 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:07.579 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/527) 2023-12-17T03:18:07.736 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:07.741 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:07.870 INFO:teuthology.orchestra.run.smithi028.stdout:generic/527 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:07.874 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/527 2023-12-17T03:18:07.874 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/527 2023-12-17T03:18:07.874 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:07.884 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:07.884 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/528) 2023-12-17T03:18:08.034 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:08.039 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:08.039 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:08.171 INFO:teuthology.orchestra.run.smithi028.stdout:generic/528 [not run] inode creation time not supported by this filesystem 2023-12-17T03:18:08.175 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/528 2023-12-17T03:18:08.175 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/528 2023-12-17T03:18:08.175 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:08.185 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:08.185 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/529) 2023-12-17T03:18:08.338 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:08.343 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:08.343 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:08.488 INFO:teuthology.orchestra.run.smithi028.stdout:generic/529 [not run] ACLs not supported by this filesystem type: ceph-fuse 2023-12-17T03:18:08.492 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/529 2023-12-17T03:18:08.492 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/529 2023-12-17T03:18:08.492 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:08.494 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:08.502 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:08.502 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/530) 2023-12-17T03:18:08.655 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:08.659 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:08.659 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:08.770 INFO:teuthology.orchestra.run.smithi028.stdout:generic/530 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:08.774 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/530 2023-12-17T03:18:08.774 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/530 2023-12-17T03:18:08.774 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:08.776 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:08.785 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:08.785 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/531) 2023-12-17T03:18:08.940 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:08.944 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:08.945 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:09.048 INFO:teuthology.orchestra.run.smithi028.stdout:generic/531 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:09.052 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/531 2023-12-17T03:18:09.052 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/531 2023-12-17T03:18:09.052 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:09.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:09.062 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:09.062 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/532) 2023-12-17T03:18:09.214 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:09.218 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:09.218 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:09.375 INFO:teuthology.orchestra.run.smithi028.stdout:generic/532 0s 2023-12-17T03:18:09.381 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/532 2023-12-17T03:18:09.381 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:09.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:09.391 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:09.391 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/533) 2023-12-17T03:18:09.545 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:09.549 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:09.549 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:09.940 INFO:teuthology.orchestra.run.smithi028.stdout:generic/533 0s 2023-12-17T03:18:09.945 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/533 2023-12-17T03:18:09.945 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:09.947 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:09.955 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:09.955 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/534) 2023-12-17T03:18:10.112 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:10.116 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:10.116 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:10.262 INFO:teuthology.orchestra.run.smithi028.stdout:generic/534 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:10.266 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/534 2023-12-17T03:18:10.266 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/534 2023-12-17T03:18:10.266 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:10.268 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:10.276 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:10.277 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/535) 2023-12-17T03:18:10.432 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:10.436 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:10.436 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:10.562 INFO:teuthology.orchestra.run.smithi028.stdout:generic/535 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:10.566 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/535 2023-12-17T03:18:10.566 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/535 2023-12-17T03:18:10.566 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:10.568 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:10.576 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:10.577 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/536) 2023-12-17T03:18:10.734 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:10.738 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:10.738 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:10.847 INFO:teuthology.orchestra.run.smithi028.stdout:generic/536 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/536 2023-12-17T03:18:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/536 2023-12-17T03:18:10.851 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:10.853 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:10.861 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:10.862 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/537) 2023-12-17T03:18:11.014 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:11.018 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:11.018 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:11.123 INFO:teuthology.orchestra.run.smithi028.stdout:generic/537 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T03:18:11.127 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/537 2023-12-17T03:18:11.127 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/537 2023-12-17T03:18:11.127 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T03:18:11.129 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T03:18:11.138 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T03:18:11.138 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/538) 2023-12-17T03:18:11.290 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T03:18:11.294 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T03:18:11.294 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:51.528 INFO:teuthology.orchestra.run.smithi028.stdout:generic/538 5560s 2023-12-17T04:50:51.586 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/538 2023-12-17T04:50:51.586 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:51.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:51.603 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:51.603 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/539) 2023-12-17T04:50:52.551 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:52.556 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:52.556 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:56.019 INFO:teuthology.orchestra.run.smithi028.stdout:generic/539 4s 2023-12-17T04:50:56.027 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/539 2023-12-17T04:50:56.027 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:56.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:56.042 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:56.042 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/540) 2023-12-17T04:50:56.611 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:56.616 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:56.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:56.748 INFO:teuthology.orchestra.run.smithi028.stdout:generic/540 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/540 2023-12-17T04:50:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/540 2023-12-17T04:50:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:56.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:56.764 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:56.764 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/541) 2023-12-17T04:50:56.932 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:56.936 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:56.936 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:57.039 INFO:teuthology.orchestra.run.smithi028.stdout:generic/541 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:57.042 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/541 2023-12-17T04:50:57.043 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/541 2023-12-17T04:50:57.043 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:57.045 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:57.053 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:57.053 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/542) 2023-12-17T04:50:57.204 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:57.208 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:57.208 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:57.310 INFO:teuthology.orchestra.run.smithi028.stdout:generic/542 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:57.313 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/542 2023-12-17T04:50:57.313 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/542 2023-12-17T04:50:57.314 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:57.316 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:57.323 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:57.324 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/543) 2023-12-17T04:50:57.472 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:57.476 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:57.588 INFO:teuthology.orchestra.run.smithi028.stdout:generic/543 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/543 2023-12-17T04:50:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/543 2023-12-17T04:50:57.592 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:57.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:57.602 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:57.602 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/544) 2023-12-17T04:50:57.757 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:57.761 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:57.866 INFO:teuthology.orchestra.run.smithi028.stdout:generic/544 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:57.870 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/544 2023-12-17T04:50:57.870 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/544 2023-12-17T04:50:57.870 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:57.872 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:57.880 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:57.880 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/545) 2023-12-17T04:50:58.028 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:58.032 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:58.032 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:58.223 INFO:teuthology.orchestra.run.smithi028.stdout:generic/545 [not run] file system doesn't support chattr +i 2023-12-17T04:50:58.226 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/545 2023-12-17T04:50:58.226 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/545 2023-12-17T04:50:58.226 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:58.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:58.237 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:58.237 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/546) 2023-12-17T04:50:58.386 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:58.390 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:58.584 INFO:teuthology.orchestra.run.smithi028.stdout:generic/546 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:58.589 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/546 2023-12-17T04:50:58.589 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/546 2023-12-17T04:50:58.589 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:58.592 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:58.600 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:58.600 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/547) 2023-12-17T04:50:58.749 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:58.753 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:58.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:58.904 INFO:teuthology.orchestra.run.smithi028.stdout:generic/547 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/547 2023-12-17T04:50:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/547 2023-12-17T04:50:58.907 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:58.909 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:58.917 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:58.918 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/548) 2023-12-17T04:50:59.070 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:59.072 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:59.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:59.211 INFO:teuthology.orchestra.run.smithi028.stdout:generic/548 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:59.215 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/548 2023-12-17T04:50:59.215 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/548 2023-12-17T04:50:59.215 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:59.217 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:59.226 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:59.226 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/549) 2023-12-17T04:50:59.382 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:59.386 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:59.387 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:59.517 INFO:teuthology.orchestra.run.smithi028.stdout:generic/549 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:59.521 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/549 2023-12-17T04:50:59.521 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/549 2023-12-17T04:50:59.521 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:59.523 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:59.531 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:59.531 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/550) 2023-12-17T04:50:59.681 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:50:59.685 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:50:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:59.817 INFO:teuthology.orchestra.run.smithi028.stdout:generic/550 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:50:59.821 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/550 2023-12-17T04:50:59.821 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/550 2023-12-17T04:50:59.822 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:50:59.824 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:50:59.832 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:50:59.832 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/551) 2023-12-17T04:50:59.996 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:00.000 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:00.000 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:00.103 INFO:teuthology.orchestra.run.smithi028.stdout:generic/551 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:00.106 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/551 2023-12-17T04:51:00.106 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/551 2023-12-17T04:51:00.107 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:00.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:00.117 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:00.117 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/552) 2023-12-17T04:51:00.268 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:00.272 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:00.272 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:00.388 INFO:teuthology.orchestra.run.smithi028.stdout:generic/552 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:00.392 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/552 2023-12-17T04:51:00.392 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/552 2023-12-17T04:51:00.392 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:00.394 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:00.402 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:00.402 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/553) 2023-12-17T04:51:00.552 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:00.556 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:00.556 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:00.730 INFO:teuthology.orchestra.run.smithi028.stdout:generic/553 [not run] xfs_io chattr +i failed (old kernel/wrong fs/bad args?) 2023-12-17T04:51:00.732 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/553 2023-12-17T04:51:00.733 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/553 2023-12-17T04:51:00.733 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:00.735 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:00.742 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:00.743 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/554) 2023-12-17T04:51:01.271 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:01.275 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:01.275 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:01.377 INFO:teuthology.orchestra.run.smithi028.stdout:generic/554 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:01.381 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/554 2023-12-17T04:51:01.381 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/554 2023-12-17T04:51:01.381 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:01.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:01.393 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:01.393 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/555) 2023-12-17T04:51:01.543 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:01.547 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:01.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:01.706 INFO:teuthology.orchestra.run.smithi028.stdout:generic/555 [not run] xfs_io chattr +ia failed (old kernel/wrong fs/bad args?) 2023-12-17T04:51:01.710 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/555 2023-12-17T04:51:01.710 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/555 2023-12-17T04:51:01.711 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:01.712 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:01.721 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:01.721 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/556) 2023-12-17T04:51:02.243 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:02.247 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:02.247 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:02.348 INFO:teuthology.orchestra.run.smithi028.stdout:generic/556 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:02.352 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/556 2023-12-17T04:51:02.352 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/556 2023-12-17T04:51:02.352 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:02.354 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:02.362 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:02.362 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/557) 2023-12-17T04:51:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:02.516 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:02.516 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:02.632 INFO:teuthology.orchestra.run.smithi028.stdout:generic/557 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:02.636 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/557 2023-12-17T04:51:02.636 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/557 2023-12-17T04:51:02.636 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:02.638 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:02.646 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:02.646 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/558) 2023-12-17T04:51:02.796 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:02.800 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:02.926 INFO:teuthology.orchestra.run.smithi028.stdout:generic/558 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:02.929 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/558 2023-12-17T04:51:02.929 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/558 2023-12-17T04:51:02.929 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:02.931 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:02.939 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:02.940 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/559) 2023-12-17T04:51:03.098 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:03.102 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:03.102 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:03.217 INFO:teuthology.orchestra.run.smithi028.stdout:generic/559 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:03.220 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/559 2023-12-17T04:51:03.220 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/559 2023-12-17T04:51:03.220 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:03.223 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:03.230 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:03.231 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/560) 2023-12-17T04:51:03.379 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:03.383 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:03.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:03.486 INFO:teuthology.orchestra.run.smithi028.stdout:generic/560 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:03.490 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/560 2023-12-17T04:51:03.490 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/560 2023-12-17T04:51:03.490 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:03.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:03.500 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:03.500 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/561) 2023-12-17T04:51:03.649 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:03.653 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:03.653 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:03.769 INFO:teuthology.orchestra.run.smithi028.stdout:generic/561 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:03.773 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/561 2023-12-17T04:51:03.773 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/561 2023-12-17T04:51:03.773 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:03.775 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:03.783 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:03.784 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/562) 2023-12-17T04:51:03.935 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:03.940 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:03.940 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:04.043 INFO:teuthology.orchestra.run.smithi028.stdout:generic/562 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:04.047 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/562 2023-12-17T04:51:04.047 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/562 2023-12-17T04:51:04.047 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:04.049 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:04.057 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:04.058 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/563) 2023-12-17T04:51:04.204 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:04.208 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:04.208 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:04.329 INFO:teuthology.orchestra.run.smithi028.stdout:generic/563 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:04.333 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/563 2023-12-17T04:51:04.334 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/563 2023-12-17T04:51:04.334 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:04.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:04.344 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:04.344 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/564) 2023-12-17T04:51:04.492 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:04.496 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:04.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:05.103 INFO:teuthology.orchestra.run.smithi028.stdout:generic/564 1s 2023-12-17T04:51:05.106 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/564 2023-12-17T04:51:05.106 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:05.118 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:05.118 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/565) 2023-12-17T04:51:05.650 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:05.654 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:05.761 INFO:teuthology.orchestra.run.smithi028.stdout:generic/565 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:05.764 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/565 2023-12-17T04:51:05.764 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/565 2023-12-17T04:51:05.765 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:05.774 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:05.775 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/566) 2023-12-17T04:51:05.923 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:05.928 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:06.029 INFO:teuthology.orchestra.run.smithi028.stdout:generic/566 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:06.033 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/566 2023-12-17T04:51:06.033 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/566 2023-12-17T04:51:06.033 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:06.043 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:06.043 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/567) 2023-12-17T04:51:06.190 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:06.194 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:06.315 INFO:teuthology.orchestra.run.smithi028.stdout:generic/567 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:06.319 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/567 2023-12-17T04:51:06.319 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/567 2023-12-17T04:51:06.319 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:06.329 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:06.329 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/568) 2023-12-17T04:51:06.481 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:06.485 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:generic/568 0s 2023-12-17T04:51:07.223 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/568 2023-12-17T04:51:07.223 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:07.259 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:07.327 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:07.327 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/569) 2023-12-17T04:51:07.511 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:07.515 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:07.516 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:07.665 INFO:teuthology.orchestra.run.smithi028.stdout:generic/569 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:07.669 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/569 2023-12-17T04:51:07.669 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/569 2023-12-17T04:51:07.669 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:07.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:07.679 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:07.679 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/570) 2023-12-17T04:51:07.829 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:07.833 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:07.833 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:07.935 INFO:teuthology.orchestra.run.smithi028.stdout:generic/570 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:07.939 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/570 2023-12-17T04:51:07.939 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/570 2023-12-17T04:51:07.940 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:07.942 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:07.950 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:07.950 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/571) 2023-12-17T04:51:08.100 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:08.104 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:08.105 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:14.352 INFO:teuthology.orchestra.run.smithi028.stdout:generic/571 6s 2023-12-17T04:51:14.359 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/571 2023-12-17T04:51:14.359 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:14.363 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:14.626 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:14.626 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/572) 2023-12-17T04:51:14.948 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:14.952 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:15.053 INFO:teuthology.orchestra.run.smithi028.stdout:generic/572 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/572 2023-12-17T04:51:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/572 2023-12-17T04:51:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:15.068 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:15.068 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/573) 2023-12-17T04:51:15.219 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:15.223 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:15.333 INFO:teuthology.orchestra.run.smithi028.stdout:generic/573 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:15.337 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/573 2023-12-17T04:51:15.337 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/573 2023-12-17T04:51:15.337 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:15.347 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:15.348 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/574) 2023-12-17T04:51:15.500 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:15.504 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:15.605 INFO:teuthology.orchestra.run.smithi028.stdout:generic/574 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:15.609 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/574 2023-12-17T04:51:15.609 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/574 2023-12-17T04:51:15.609 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:15.619 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:15.619 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/575) 2023-12-17T04:51:15.769 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:15.773 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:15.873 INFO:teuthology.orchestra.run.smithi028.stdout:generic/575 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:15.877 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/575 2023-12-17T04:51:15.877 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/575 2023-12-17T04:51:15.877 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:15.887 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:15.887 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/576) 2023-12-17T04:51:16.034 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:16.039 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:16.177 INFO:teuthology.orchestra.run.smithi028.stdout:generic/576 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:16.180 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/576 2023-12-17T04:51:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/576 2023-12-17T04:51:16.181 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:16.191 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:16.191 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/577) 2023-12-17T04:51:16.344 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:16.348 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:16.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:16.522 INFO:teuthology.orchestra.run.smithi028.stdout:generic/577 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:16.526 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/577 2023-12-17T04:51:16.526 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/577 2023-12-17T04:51:16.526 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:16.528 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:16.537 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:16.537 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/578) 2023-12-17T04:51:16.689 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:16.693 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:16.693 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:16.947 INFO:teuthology.orchestra.run.smithi028.stdout:generic/578 [not run] xfs_io fiemap failed (old kernel/wrong fs?) 2023-12-17T04:51:16.951 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/578 2023-12-17T04:51:16.951 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/578 2023-12-17T04:51:16.952 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:16.954 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:16.963 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:16.963 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/579) 2023-12-17T04:51:17.123 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:17.128 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:17.128 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:generic/579 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/579 2023-12-17T04:51:17.254 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/579 2023-12-17T04:51:17.255 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:17.257 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:17.266 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:17.267 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/580) 2023-12-17T04:51:17.425 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:17.429 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:17.429 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:17.570 INFO:teuthology.orchestra.run.smithi028.stdout:generic/580 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:17.575 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/580 2023-12-17T04:51:17.575 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/580 2023-12-17T04:51:17.575 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:17.577 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:17.587 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:17.587 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/581) 2023-12-17T04:51:17.741 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:17.745 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:17.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:18.301 INFO:teuthology.orchestra.run.smithi028.stdout:generic/581 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:18.306 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/581 2023-12-17T04:51:18.306 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/581 2023-12-17T04:51:18.306 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:18.310 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:18.321 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:18.322 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/582) 2023-12-17T04:51:18.517 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:18.522 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:18.523 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:generic/582 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:18.667 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/582 2023-12-17T04:51:18.667 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/582 2023-12-17T04:51:18.668 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:18.678 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:18.678 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/583) 2023-12-17T04:51:18.835 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:18.839 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:18.839 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:18.975 INFO:teuthology.orchestra.run.smithi028.stdout:generic/583 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:18.979 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/583 2023-12-17T04:51:18.979 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/583 2023-12-17T04:51:18.980 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:18.982 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:18.990 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:18.990 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/584) 2023-12-17T04:51:19.145 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:19.149 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:19.149 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:19.290 INFO:teuthology.orchestra.run.smithi028.stdout:generic/584 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/584 2023-12-17T04:51:19.295 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/584 2023-12-17T04:51:19.295 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:19.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:19.305 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:19.305 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/585) 2023-12-17T04:51:19.461 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:19.465 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:19.465 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:19.583 INFO:teuthology.orchestra.run.smithi028.stdout:generic/585 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:51:19.587 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/585 2023-12-17T04:51:19.587 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/585 2023-12-17T04:51:19.587 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:51:19.589 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:51:19.597 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:51:19.597 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/586) 2023-12-17T04:51:19.747 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:51:19.752 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:51:19.752 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:08.001 INFO:teuthology.orchestra.run.smithi028.stdout:generic/586 48s 2023-12-17T04:52:08.006 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/586 2023-12-17T04:52:08.006 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:08.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:08.025 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:08.027 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/587) 2023-12-17T04:52:08.588 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:08.592 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:08.917 INFO:teuthology.orchestra.run.smithi028.stdout:generic/587 [not run] disk quotas not supported by this filesystem type: ceph-fuse 2023-12-17T04:52:08.921 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/587 2023-12-17T04:52:08.921 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/587 2023-12-17T04:52:08.921 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:08.924 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:08.933 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:08.933 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/588) 2023-12-17T04:52:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:09.103 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:09.274 INFO:teuthology.orchestra.run.smithi028.stdout:generic/588 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:09.278 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/588 2023-12-17T04:52:09.278 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/588 2023-12-17T04:52:09.278 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:09.281 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:09.289 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:09.290 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/589) 2023-12-17T04:52:09.447 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:09.449 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:09.450 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:09.559 INFO:teuthology.orchestra.run.smithi028.stdout:generic/589 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:09.562 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/589 2023-12-17T04:52:09.562 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/589 2023-12-17T04:52:09.562 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:09.565 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:09.573 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:09.573 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/590) 2023-12-17T04:52:09.730 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:09.734 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:09.735 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:09.840 INFO:teuthology.orchestra.run.smithi028.stdout:generic/590 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:09.844 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/590 2023-12-17T04:52:09.844 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/590 2023-12-17T04:52:09.844 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:09.846 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:09.854 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:09.854 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/591) 2023-12-17T04:52:10.003 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:10.007 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:10.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:10.406 INFO:teuthology.orchestra.run.smithi028.stdout:generic/591 0s 2023-12-17T04:52:10.413 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/591 2023-12-17T04:52:10.413 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:10.417 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:10.429 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:10.429 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/592) 2023-12-17T04:52:10.964 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:10.967 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:10.967 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:11.100 INFO:teuthology.orchestra.run.smithi028.stdout:generic/592 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:11.103 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/592 2023-12-17T04:52:11.103 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/592 2023-12-17T04:52:11.104 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:11.106 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:11.114 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:11.114 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/593) 2023-12-17T04:52:11.262 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:11.266 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:11.266 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:11.394 INFO:teuthology.orchestra.run.smithi028.stdout:generic/593 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:11.398 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/593 2023-12-17T04:52:11.398 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/593 2023-12-17T04:52:11.398 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:11.400 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:11.408 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:11.409 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/594) 2023-12-17T04:52:11.562 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:11.567 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:11.567 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:11.671 INFO:teuthology.orchestra.run.smithi028.stdout:generic/594 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:11.675 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/594 2023-12-17T04:52:11.675 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/594 2023-12-17T04:52:11.675 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:11.677 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:11.684 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:11.684 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/595) 2023-12-17T04:52:11.834 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:11.838 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:11.838 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:11.978 INFO:teuthology.orchestra.run.smithi028.stdout:generic/595 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:11.982 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/595 2023-12-17T04:52:11.982 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/595 2023-12-17T04:52:11.982 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:11.985 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:11.992 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:11.992 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/596) 2023-12-17T04:52:12.143 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:12.147 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:12.148 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:12.252 INFO:teuthology.orchestra.run.smithi028.stdout:generic/596 [not run] accton utility required, skipped this test 2023-12-17T04:52:12.255 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/596 2023-12-17T04:52:12.255 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/596 2023-12-17T04:52:12.255 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:12.258 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:12.265 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:12.266 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/597) 2023-12-17T04:52:12.416 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:12.420 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:12.420 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:12.906 INFO:teuthology.orchestra.run.smithi028.stdout:generic/597 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/597.out.bad) 2023-12-17T04:52:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/597.out 2023-12-16 21:58:54.289019006 +0000 2023-12-17T04:52:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/597.out.bad 2023-12-17 04:52:12.904010625 +0000 2023-12-17T04:52:12.908 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -4,11 +4,11 @@ 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_symlinks = 0 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: successfully followed symlink 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_symlinks = 1 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: -Permission denied 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: +successfully followed symlink 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: == Test hardlink create protection when 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: == process != target owner and process cannot read target 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_hardlinks = 0 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: -successfully created hardlink 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: +Permission denied 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_hardlinks = 1 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: -ln: failed to create hard link 'TEST_DIR/597/sticky_dir/hardlink' => 'TEST_DIR/597/target': Operation not permitted 2023-12-17T04:52:12.909 INFO:teuthology.orchestra.run.smithi028.stdout: +Permission denied 2023-12-17T04:52:12.911 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/597 2023-12-17T04:52:12.911 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/597 2023-12-17T04:52:12.911 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T04:52:12.913 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T04:52:12.921 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T04:52:12.921 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T04:52:12.922 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/598) 2023-12-17T04:52:13.423 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:13.427 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:13.427 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:13.884 INFO:teuthology.orchestra.run.smithi028.stdout:generic/598 - output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/598.out.bad) 2023-12-17T04:52:13.886 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/598.out 2023-12-16 21:58:54.289019006 +0000 2023-12-17T04:52:13.886 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/598.out.bad 2023-12-17 04:52:13.879982196 +0000 2023-12-17T04:52:13.886 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -5,13 +5,10 @@ 2023-12-17T04:52:13.886 INFO:teuthology.orchestra.run.smithi028.stdout: = only group writable dir 2023-12-17T04:52:13.886 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_regular = 1 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: = group & world writable dir 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: -Permission denied 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: = only group writable dir 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_regular = 2 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: = group & world writable dir 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: -Permission denied 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: = only group writable dir 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: -Permission denied 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: == Test fifo open when owned by another and fifo owner != dir owner 2023-12-17T04:52:13.888 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_fifos = 0 2023-12-17T04:52:13.888 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -19,10 +16,7 @@ 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: = only group writable dir 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_fifos = 1 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: = group & world writable dir 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: -Permission denied 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: = only group writable dir 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: fs.protected_fifos = 2 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: = group & world writable dir 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: -Permission denied 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: = only group writable dir 2023-12-17T04:52:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: -Permission denied 2023-12-17T04:52:13.890 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/598 2023-12-17T04:52:13.890 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/598 2023-12-17T04:52:13.890 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T04:52:13.891 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:13.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T04:52:13.900 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T04:52:13.900 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T04:52:13.900 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/599) 2023-12-17T04:52:14.388 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:14.392 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:14.392 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:14.497 INFO:teuthology.orchestra.run.smithi028.stdout:generic/599 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:14.500 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/599 2023-12-17T04:52:14.500 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/599 2023-12-17T04:52:14.500 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:14.502 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:14.511 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:14.511 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/600) 2023-12-17T04:52:14.660 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:14.665 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:14.665 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:14.769 INFO:teuthology.orchestra.run.smithi028.stdout:generic/600 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:14.772 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/600 2023-12-17T04:52:14.772 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/600 2023-12-17T04:52:14.772 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:14.775 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:14.782 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:14.783 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/601) 2023-12-17T04:52:14.932 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:14.936 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:15.039 INFO:teuthology.orchestra.run.smithi028.stdout:generic/601 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:15.043 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/601 2023-12-17T04:52:15.043 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/601 2023-12-17T04:52:15.043 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:15.053 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:15.053 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/602) 2023-12-17T04:52:15.202 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:15.206 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:15.342 INFO:teuthology.orchestra.run.smithi028.stdout:generic/602 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:15.346 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/602 2023-12-17T04:52:15.346 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/602 2023-12-17T04:52:15.346 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:15.357 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:15.357 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/603) 2023-12-17T04:52:15.511 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:15.515 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:15.639 INFO:teuthology.orchestra.run.smithi028.stdout:generic/603 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:15.643 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/603 2023-12-17T04:52:15.643 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/603 2023-12-17T04:52:15.643 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:15.655 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:15.655 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/604) 2023-12-17T04:52:15.811 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:15.815 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:15.921 INFO:teuthology.orchestra.run.smithi028.stdout:generic/604 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:15.924 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/604 2023-12-17T04:52:15.924 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/604 2023-12-17T04:52:15.924 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:15.934 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:15.934 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/605) 2023-12-17T04:52:16.084 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:16.088 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:16.192 INFO:teuthology.orchestra.run.smithi028.stdout:generic/605 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:16.196 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/605 2023-12-17T04:52:16.196 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/605 2023-12-17T04:52:16.196 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:16.206 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:16.206 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/606) 2023-12-17T04:52:16.354 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:16.358 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:16.358 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:16.461 INFO:teuthology.orchestra.run.smithi028.stdout:generic/606 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:16.465 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/606 2023-12-17T04:52:16.465 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/606 2023-12-17T04:52:16.465 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:16.467 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:16.475 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:16.476 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/607) 2023-12-17T04:52:16.626 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:16.631 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:16.631 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout:generic/607 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/607 2023-12-17T04:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/607 2023-12-17T04:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:16.747 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:16.747 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/608) 2023-12-17T04:52:16.899 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:16.903 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:16.903 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:17.006 INFO:teuthology.orchestra.run.smithi028.stdout:generic/608 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/608 2023-12-17T04:52:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/608 2023-12-17T04:52:17.010 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:17.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:17.020 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:17.020 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/609) 2023-12-17T04:52:17.171 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:17.175 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:17.756 INFO:teuthology.orchestra.run.smithi028.stdout:generic/609 0s 2023-12-17T04:52:17.765 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/609 2023-12-17T04:52:17.765 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:17.769 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:17.781 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:17.781 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/610) 2023-12-17T04:52:18.316 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:18.320 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:18.424 INFO:teuthology.orchestra.run.smithi028.stdout:generic/610 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:18.428 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/610 2023-12-17T04:52:18.428 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/610 2023-12-17T04:52:18.428 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:18.439 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:18.439 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/611) 2023-12-17T04:52:18.590 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:18.594 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:18.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:18.707 INFO:teuthology.orchestra.run.smithi028.stdout:generic/611 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:18.711 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/611 2023-12-17T04:52:18.711 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/611 2023-12-17T04:52:18.711 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:18.714 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:18.723 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:18.723 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/612) 2023-12-17T04:52:18.884 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:18.888 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:18.888 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:19.047 INFO:teuthology.orchestra.run.smithi028.stdout:generic/612 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-17T04:52:19.051 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/612 2023-12-17T04:52:19.051 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/612 2023-12-17T04:52:19.051 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:19.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:19.061 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:19.061 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/613) 2023-12-17T04:52:19.562 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:19.566 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:19.698 INFO:teuthology.orchestra.run.smithi028.stdout:generic/613 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:19.702 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/613 2023-12-17T04:52:19.702 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/613 2023-12-17T04:52:19.702 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:19.705 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:19.713 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:19.713 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/614) 2023-12-17T04:52:19.867 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:19.871 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:19.871 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:19.975 INFO:teuthology.orchestra.run.smithi028.stdout:generic/614 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/614 2023-12-17T04:52:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/614 2023-12-17T04:52:19.979 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:19.981 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:19.989 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:19.989 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/615) 2023-12-17T04:52:20.139 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:20.143 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:20.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:20.256 INFO:teuthology.orchestra.run.smithi028.stdout:generic/615 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T04:52:20.259 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/615 2023-12-17T04:52:20.259 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/615 2023-12-17T04:52:20.259 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T04:52:20.262 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T04:52:20.270 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T04:52:20.270 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/616) 2023-12-17T04:52:20.424 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T04:52:20.427 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T04:52:20.427 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:00:17.437 INFO:teuthology.orchestra.run.smithi028.stdout:generic/616 477s 2023-12-17T05:00:17.444 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/616 2023-12-17T05:00:17.444 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:00:17.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:00:17.461 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:00:17.461 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/617) 2023-12-17T05:00:17.664 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:00:17.670 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:00:17.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:48.315 INFO:teuthology.orchestra.run.smithi028.stdout:generic/617 151s 2023-12-17T05:02:48.317 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/617 2023-12-17T05:02:48.318 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:48.318 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:48.555 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:48.558 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/618) 2023-12-17T05:02:48.902 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:48.906 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:48.906 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:49.017 INFO:teuthology.orchestra.run.smithi028.stdout:generic/618 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:49.020 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/618 2023-12-17T05:02:49.021 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/618 2023-12-17T05:02:49.021 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:49.023 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:49.031 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:49.032 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/619) 2023-12-17T05:02:49.185 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:49.189 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:49.189 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:49.290 INFO:teuthology.orchestra.run.smithi028.stdout:generic/619 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:49.294 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/619 2023-12-17T05:02:49.294 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/619 2023-12-17T05:02:49.294 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:49.296 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:49.303 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:49.304 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/620) 2023-12-17T05:02:49.456 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:49.460 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:49.461 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:generic/620 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:49.762 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/620 2023-12-17T05:02:49.762 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/620 2023-12-17T05:02:49.762 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:49.765 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:49.776 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:49.777 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/621) 2023-12-17T05:02:49.946 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:49.950 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:49.950 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:50.082 INFO:teuthology.orchestra.run.smithi028.stdout:generic/621 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:50.086 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/621 2023-12-17T05:02:50.086 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/621 2023-12-17T05:02:50.086 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:50.088 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:50.096 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:50.097 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/622) 2023-12-17T05:02:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:50.256 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:50.256 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:50.369 INFO:teuthology.orchestra.run.smithi028.stdout:generic/622 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:50.373 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/622 2023-12-17T05:02:50.373 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/622 2023-12-17T05:02:50.373 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:50.375 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:50.383 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:50.383 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/623) 2023-12-17T05:02:50.537 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:50.541 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:50.541 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:50.643 INFO:teuthology.orchestra.run.smithi028.stdout:generic/623 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:50.647 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/623 2023-12-17T05:02:50.647 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/623 2023-12-17T05:02:50.647 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:50.649 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:50.657 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:50.657 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/624) 2023-12-17T05:02:50.812 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:50.816 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:50.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:50.919 INFO:teuthology.orchestra.run.smithi028.stdout:generic/624 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:50.923 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/624 2023-12-17T05:02:50.923 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/624 2023-12-17T05:02:50.924 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:50.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:50.934 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:50.934 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/625) 2023-12-17T05:02:51.095 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:51.099 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:51.099 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:51.202 INFO:teuthology.orchestra.run.smithi028.stdout:generic/625 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:51.206 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/625 2023-12-17T05:02:51.206 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/625 2023-12-17T05:02:51.206 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:51.208 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:51.216 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:51.216 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/626) 2023-12-17T05:02:51.368 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:51.372 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:51.372 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:51.475 INFO:teuthology.orchestra.run.smithi028.stdout:generic/626 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:51.479 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/626 2023-12-17T05:02:51.479 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/626 2023-12-17T05:02:51.479 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:51.481 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:51.489 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:51.489 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/627) 2023-12-17T05:02:51.639 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:51.643 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:51.643 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:51.751 INFO:teuthology.orchestra.run.smithi028.stdout:generic/627 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:51.755 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/627 2023-12-17T05:02:51.755 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/627 2023-12-17T05:02:51.755 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:51.757 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:51.766 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:51.766 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/628) 2023-12-17T05:02:51.918 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:51.922 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:51.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:52.056 INFO:teuthology.orchestra.run.smithi028.stdout:generic/628 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:52.059 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/628 2023-12-17T05:02:52.060 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/628 2023-12-17T05:02:52.060 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:52.070 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:52.070 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/629) 2023-12-17T05:02:52.221 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:52.225 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:52.225 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:52.340 INFO:teuthology.orchestra.run.smithi028.stdout:generic/629 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:52.344 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/629 2023-12-17T05:02:52.344 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/629 2023-12-17T05:02:52.344 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:52.346 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:52.354 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:52.354 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/630) 2023-12-17T05:02:52.508 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:52.512 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:52.512 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:52.620 INFO:teuthology.orchestra.run.smithi028.stdout:generic/630 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:52.625 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/630 2023-12-17T05:02:52.625 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/630 2023-12-17T05:02:52.625 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:52.626 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:52.635 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:52.635 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/631) 2023-12-17T05:02:52.786 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:52.790 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:52.790 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:52.900 INFO:teuthology.orchestra.run.smithi028.stdout:generic/631 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:02:52.904 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/631 2023-12-17T05:02:52.904 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/631 2023-12-17T05:02:52.905 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:02:52.907 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:02:52.915 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:02:52.915 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/632) 2023-12-17T05:02:53.068 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:02:53.072 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:02:53.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:00.454 INFO:teuthology.orchestra.run.smithi028.stdout:generic/632 7s 2023-12-17T05:03:00.461 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/632 2023-12-17T05:03:00.462 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:00.465 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:00.524 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:00.524 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/633) 2023-12-17T05:03:00.727 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:00.732 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:00.732 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:01.237 INFO:teuthology.orchestra.run.smithi028.stdout:generic/633 [failed, exit status 1]- output mismatch (see /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/633.out.bad) 2023-12-17T05:03:01.239 INFO:teuthology.orchestra.run.smithi028.stdout: --- tests/generic/633.out 2023-12-16 21:58:54.289019006 +0000 2023-12-17T05:03:01.239 INFO:teuthology.orchestra.run.smithi028.stdout: +++ /tmp/tmp.nC0MjoAvrDxfstests-dev/results//generic/633.out.bad 2023-12-17 05:03:01.229130087 +0000 2023-12-17T05:03:01.239 INFO:teuthology.orchestra.run.smithi028.stdout: @@ -1,2 +1,5 @@ 2023-12-17T05:03:01.240 INFO:teuthology.orchestra.run.smithi028.stdout: QA output created by 633 2023-12-17T05:03:01.240 INFO:teuthology.orchestra.run.smithi028.stdout: Silence is golden 2023-12-17T05:03:01.240 INFO:teuthology.orchestra.run.smithi028.stdout: +vfstest.c: 1164: protected_symlinks - Success - failure: openat 2023-12-17T05:03:01.240 INFO:teuthology.orchestra.run.smithi028.stdout: +vfstest.c: 1171: protected_symlinks - Success - failure: wait_for_pid 2023-12-17T05:03:01.240 INFO:teuthology.orchestra.run.smithi028.stdout: +vfstest.c: 2418: run_test - Success - failure: following protected symlinks on regular mounts 2023-12-17T05:03:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/633 2023-12-17T05:03:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:Failures: generic/633 2023-12-17T05:03:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:Failed 1 of 1 tests 2023-12-17T05:03:01.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T05:03:01.264 INFO:tasks.cephfs.xfstests_dev:Command failed 2023-12-17T05:03:01.264 INFO:tasks.cephfs.xfstests_dev:Command return value: 1 2023-12-17T05:03:01.265 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/634) 2023-12-17T05:03:01.770 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:01.774 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:01.774 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:01.875 INFO:teuthology.orchestra.run.smithi028.stdout:generic/634 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:01.878 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/634 2023-12-17T05:03:01.879 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/634 2023-12-17T05:03:01.879 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:01.888 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:01.889 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/635) 2023-12-17T05:03:02.054 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:02.059 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:02.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:02.181 INFO:teuthology.orchestra.run.smithi028.stdout:generic/635 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:02.185 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/635 2023-12-17T05:03:02.185 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/635 2023-12-17T05:03:02.185 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:02.195 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:02.195 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/636) 2023-12-17T05:03:02.348 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:02.351 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:02.351 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:02.475 INFO:teuthology.orchestra.run.smithi028.stdout:generic/636 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:02.479 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/636 2023-12-17T05:03:02.479 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/636 2023-12-17T05:03:02.479 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:02.481 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:02.489 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:02.490 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/637) 2023-12-17T05:03:02.640 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:02.644 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:02.644 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:04.630 INFO:teuthology.orchestra.run.smithi028.stdout:generic/637 2s 2023-12-17T05:03:04.638 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/637 2023-12-17T05:03:04.638 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:04.642 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:04.692 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:04.692 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/638) 2023-12-17T05:03:05.270 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:05.274 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:05.274 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:05.446 INFO:teuthology.orchestra.run.smithi028.stdout:generic/638 0s 2023-12-17T05:03:05.451 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/638 2023-12-17T05:03:05.451 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:05.461 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:05.462 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/639) 2023-12-17T05:03:05.616 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:05.620 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:10.346 INFO:teuthology.orchestra.run.smithi028.stdout:generic/639 5s 2023-12-17T05:03:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/639 2023-12-17T05:03:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:10.358 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:10.368 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:10.369 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/640) 2023-12-17T05:03:10.568 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:10.573 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:10.573 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:10.749 INFO:teuthology.orchestra.run.smithi028.stdout:generic/640 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:10.754 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/640 2023-12-17T05:03:10.754 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/640 2023-12-17T05:03:10.754 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:10.757 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:10.767 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:10.768 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/641) 2023-12-17T05:03:10.961 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:10.966 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:10.966 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:11.076 INFO:teuthology.orchestra.run.smithi028.stdout:generic/641 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:11.080 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/641 2023-12-17T05:03:11.080 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/641 2023-12-17T05:03:11.080 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:11.082 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:11.090 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:11.090 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/642) 2023-12-17T05:03:11.248 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:11.252 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:11.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:11.368 INFO:teuthology.orchestra.run.smithi028.stdout:generic/642 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:11.372 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/642 2023-12-17T05:03:11.372 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/642 2023-12-17T05:03:11.372 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:11.383 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:11.383 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/643) 2023-12-17T05:03:11.536 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:11.540 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:11.540 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:11.643 INFO:teuthology.orchestra.run.smithi028.stdout:generic/643 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:11.647 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/643 2023-12-17T05:03:11.647 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/643 2023-12-17T05:03:11.647 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:11.649 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:11.657 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:11.658 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/644) 2023-12-17T05:03:11.816 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:11.820 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:11.820 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:11.963 INFO:teuthology.orchestra.run.smithi028.stdout:generic/644 [not run] vfstest not support by ceph-fuse 2023-12-17T05:03:11.967 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/644 2023-12-17T05:03:11.967 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/644 2023-12-17T05:03:11.967 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:11.969 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:11.977 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:11.977 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/645) 2023-12-17T05:03:12.132 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:12.136 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:12.136 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:12.239 INFO:teuthology.orchestra.run.smithi028.stdout:generic/645 [not run] vfstest not support by ceph-fuse 2023-12-17T05:03:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/645 2023-12-17T05:03:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/645 2023-12-17T05:03:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:12.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:12.252 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:12.253 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/646) 2023-12-17T05:03:12.407 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:12.411 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:12.412 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:12.515 INFO:teuthology.orchestra.run.smithi028.stdout:generic/646 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:12.519 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/646 2023-12-17T05:03:12.519 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/646 2023-12-17T05:03:12.519 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:12.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:12.529 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:12.529 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/647) 2023-12-17T05:03:12.682 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:12.686 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:12.686 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:13.123 INFO:teuthology.orchestra.run.smithi028.stdout:generic/647 1s 2023-12-17T05:03:13.131 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/647 2023-12-17T05:03:13.131 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:13.135 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:13.145 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:13.145 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/648) 2023-12-17T05:03:13.669 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:13.673 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:13.673 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:13.812 INFO:teuthology.orchestra.run.smithi028.stdout:generic/648 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T05:03:13.816 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/648 2023-12-17T05:03:13.816 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/648 2023-12-17T05:03:13.816 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:13.818 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:13.826 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:13.826 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/649) 2023-12-17T05:03:13.980 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:13.984 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:14.277 INFO:teuthology.orchestra.run.smithi028.stdout:generic/649 [not run] Reflink not supported by test filesystem type: ceph-fuse 2023-12-17T05:03:14.281 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/649 2023-12-17T05:03:14.282 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/649 2023-12-17T05:03:14.282 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T05:03:14.284 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T05:03:14.293 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T05:03:14.293 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/650) 2023-12-17T05:03:14.798 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T05:03:14.802 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T05:03:14.802 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:28.380 INFO:teuthology.orchestra.run.smithi028.stdout:generic/650 3614s 2023-12-17T06:03:28.433 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/650 2023-12-17T06:03:28.433 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:28.447 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:28.448 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/651) 2023-12-17T06:03:29.046 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:29.052 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:29.241 INFO:teuthology.orchestra.run.smithi028.stdout:generic/651 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:29.246 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/651 2023-12-17T06:03:29.246 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/651 2023-12-17T06:03:29.246 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:29.249 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:29.263 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:29.264 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/652) 2023-12-17T06:03:29.507 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:29.514 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:29.514 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:29.678 INFO:teuthology.orchestra.run.smithi028.stdout:generic/652 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:29.682 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/652 2023-12-17T06:03:29.682 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/652 2023-12-17T06:03:29.682 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:29.685 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:29.701 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:29.701 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/653) 2023-12-17T06:03:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:29.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:30.143 INFO:teuthology.orchestra.run.smithi028.stdout:generic/653 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:30.147 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/653 2023-12-17T06:03:30.147 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/653 2023-12-17T06:03:30.147 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:30.169 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:30.170 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/654) 2023-12-17T06:03:30.440 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:30.445 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:30.445 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:30.774 INFO:teuthology.orchestra.run.smithi028.stdout:generic/654 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:30.779 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/654 2023-12-17T06:03:30.779 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/654 2023-12-17T06:03:30.779 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:30.781 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:30.795 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:30.795 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/655) 2023-12-17T06:03:31.038 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:31.045 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:31.045 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:31.212 INFO:teuthology.orchestra.run.smithi028.stdout:generic/655 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:31.217 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/655 2023-12-17T06:03:31.217 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/655 2023-12-17T06:03:31.217 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:31.220 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:31.232 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:31.232 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/656) 2023-12-17T06:03:31.486 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:31.496 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:31.655 INFO:teuthology.orchestra.run.smithi028.stdout:generic/656 [not run] vfstest not support by ceph-fuse 2023-12-17T06:03:31.659 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/656 2023-12-17T06:03:31.659 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/656 2023-12-17T06:03:31.659 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:31.662 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:31.678 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:31.679 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/657) 2023-12-17T06:03:31.938 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:31.951 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:31.951 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:32.183 INFO:teuthology.orchestra.run.smithi028.stdout:generic/657 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:32.187 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/657 2023-12-17T06:03:32.187 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/657 2023-12-17T06:03:32.187 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:32.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:32.213 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:32.213 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/658) 2023-12-17T06:03:32.454 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:32.458 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:32.458 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:32.916 INFO:teuthology.orchestra.run.smithi028.stdout:generic/658 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:32.920 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/658 2023-12-17T06:03:32.920 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/658 2023-12-17T06:03:32.921 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:32.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:32.940 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:32.940 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/659) 2023-12-17T06:03:33.247 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:33.253 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:33.253 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:34.378 INFO:teuthology.orchestra.run.smithi028.stdout:generic/659 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/659 2023-12-17T06:03:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/659 2023-12-17T06:03:34.383 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:34.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:34.394 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:34.395 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/660) 2023-12-17T06:03:34.640 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:34.652 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:34.652 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:35.407 INFO:teuthology.orchestra.run.smithi028.stdout:generic/660 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:35.411 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/660 2023-12-17T06:03:35.411 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/660 2023-12-17T06:03:35.411 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:35.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:35.429 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:35.430 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/661) 2023-12-17T06:03:35.704 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:35.709 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:35.709 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:35.925 INFO:teuthology.orchestra.run.smithi028.stdout:generic/661 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/661 2023-12-17T06:03:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/661 2023-12-17T06:03:35.930 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:35.943 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:35.952 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:35.952 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/662) 2023-12-17T06:03:36.211 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:36.217 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:36.217 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:36.712 INFO:teuthology.orchestra.run.smithi028.stdout:generic/662 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:36.716 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/662 2023-12-17T06:03:36.716 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/662 2023-12-17T06:03:36.717 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:36.719 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:36.737 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:36.738 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/663) 2023-12-17T06:03:36.991 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:36.996 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:36.996 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:37.242 INFO:teuthology.orchestra.run.smithi028.stdout:generic/663 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:37.247 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/663 2023-12-17T06:03:37.248 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/663 2023-12-17T06:03:37.248 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:37.264 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:37.264 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/664) 2023-12-17T06:03:37.518 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:37.529 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:37.707 INFO:teuthology.orchestra.run.smithi028.stdout:generic/664 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:37.711 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/664 2023-12-17T06:03:37.712 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/664 2023-12-17T06:03:37.712 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:37.714 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:37.731 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:37.731 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/665) 2023-12-17T06:03:37.971 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:37.976 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:37.976 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:38.142 INFO:teuthology.orchestra.run.smithi028.stdout:generic/665 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:38.156 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/665 2023-12-17T06:03:38.156 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/665 2023-12-17T06:03:38.156 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:38.159 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:38.169 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:38.169 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/666) 2023-12-17T06:03:38.436 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:38.446 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:38.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:38.620 INFO:teuthology.orchestra.run.smithi028.stdout:generic/666 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:38.625 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/666 2023-12-17T06:03:38.626 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/666 2023-12-17T06:03:38.626 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:38.629 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:38.644 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:38.645 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/667) 2023-12-17T06:03:38.883 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:38.889 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:38.890 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:39.069 INFO:teuthology.orchestra.run.smithi028.stdout:generic/667 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:39.073 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/667 2023-12-17T06:03:39.073 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/667 2023-12-17T06:03:39.073 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:39.075 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:39.085 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:39.086 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/668) 2023-12-17T06:03:39.379 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:39.388 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:39.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:39.543 INFO:teuthology.orchestra.run.smithi028.stdout:generic/668 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:39.547 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/668 2023-12-17T06:03:39.547 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/668 2023-12-17T06:03:39.547 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:39.550 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:39.560 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:39.561 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/669) 2023-12-17T06:03:39.799 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:39.804 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:39.974 INFO:teuthology.orchestra.run.smithi028.stdout:generic/669 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:39.979 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/669 2023-12-17T06:03:39.979 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/669 2023-12-17T06:03:39.979 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:39.983 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:39.993 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:39.994 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/670) 2023-12-17T06:03:40.290 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:40.295 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:40.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:40.473 INFO:teuthology.orchestra.run.smithi028.stdout:generic/670 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:40.477 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/670 2023-12-17T06:03:40.477 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/670 2023-12-17T06:03:40.477 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:40.482 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:40.497 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:40.497 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/671) 2023-12-17T06:03:40.758 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:40.764 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:40.764 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:40.946 INFO:teuthology.orchestra.run.smithi028.stdout:generic/671 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:40.954 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/671 2023-12-17T06:03:40.954 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/671 2023-12-17T06:03:40.954 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:40.962 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:40.975 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:40.975 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/672) 2023-12-17T06:03:41.240 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:41.245 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:41.246 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:41.408 INFO:teuthology.orchestra.run.smithi028.stdout:generic/672 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:41.413 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/672 2023-12-17T06:03:41.413 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/672 2023-12-17T06:03:41.413 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:41.416 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:41.426 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:41.426 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/673) 2023-12-17T06:03:41.667 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:41.671 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:41.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:42.180 INFO:teuthology.orchestra.run.smithi028.stdout:generic/673 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:42.185 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/673 2023-12-17T06:03:42.185 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/673 2023-12-17T06:03:42.185 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:42.197 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:42.198 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/674) 2023-12-17T06:03:42.496 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:42.501 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:42.671 INFO:teuthology.orchestra.run.smithi028.stdout:generic/674 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:42.675 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/674 2023-12-17T06:03:42.675 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/674 2023-12-17T06:03:42.676 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:42.693 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:42.693 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/675) 2023-12-17T06:03:42.931 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:42.935 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:43.207 INFO:teuthology.orchestra.run.smithi028.stdout:generic/675 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:03:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/675 2023-12-17T06:03:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/675 2023-12-17T06:03:43.212 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:03:43.214 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:03:43.233 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:03:43.233 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/676) 2023-12-17T06:03:43.485 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:03:43.495 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:03:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:06.830 INFO:teuthology.orchestra.run.smithi028.stdout:generic/676 1643s 2023-12-17T06:31:06.883 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/676 2023-12-17T06:31:06.884 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:06.900 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:06.900 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/677) 2023-12-17T06:31:07.720 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:07.724 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:07.724 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:07.974 INFO:teuthology.orchestra.run.smithi028.stdout:generic/677 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:07.981 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/677 2023-12-17T06:31:07.981 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/677 2023-12-17T06:31:07.981 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:07.989 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:07.997 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:07.998 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/678) 2023-12-17T06:31:08.299 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:08.304 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:08.304 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:08.724 INFO:teuthology.orchestra.run.smithi028.stdout:generic/678 0s 2023-12-17T06:31:08.730 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/678 2023-12-17T06:31:08.731 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:08.733 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:08.744 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:08.744 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/679) 2023-12-17T06:31:09.362 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:09.367 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:09.367 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:09.948 INFO:teuthology.orchestra.run.smithi028.stdout:generic/679 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:09.952 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/679 2023-12-17T06:31:09.953 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/679 2023-12-17T06:31:09.953 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:09.956 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:09.967 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:09.968 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/680) 2023-12-17T06:31:10.247 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:10.252 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:10.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:12.401 INFO:teuthology.orchestra.run.smithi028.stdout:generic/680 2s 2023-12-17T06:31:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/680 2023-12-17T06:31:12.410 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:12.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:12.427 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:12.428 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/681) 2023-12-17T06:31:13.079 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:13.084 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:13.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:13.342 INFO:teuthology.orchestra.run.smithi028.stdout:generic/681 [not run] disk quotas not supported by this filesystem type: ceph-fuse 2023-12-17T06:31:13.350 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/681 2023-12-17T06:31:13.350 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/681 2023-12-17T06:31:13.350 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:13.352 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:13.361 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:13.361 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/682) 2023-12-17T06:31:13.611 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:13.616 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:13.798 INFO:teuthology.orchestra.run.smithi028.stdout:generic/682 [not run] disk quotas not supported by this filesystem type: ceph-fuse 2023-12-17T06:31:13.802 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/682 2023-12-17T06:31:13.802 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/682 2023-12-17T06:31:13.802 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:13.812 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:13.839 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:13.839 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/683) 2023-12-17T06:31:14.086 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:14.091 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:14.091 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:14.593 INFO:teuthology.orchestra.run.smithi028.stdout:generic/683 [not run] /tmp/tmp.XR9uc7m4Witest: file alloc unit 4194304 larger than op length 65536 2023-12-17T06:31:14.598 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/683 2023-12-17T06:31:14.598 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/683 2023-12-17T06:31:14.598 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:14.600 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:14.610 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:14.610 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/684) 2023-12-17T06:31:14.813 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:14.819 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:14.819 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:16.787 INFO:teuthology.orchestra.run.smithi028.stdout:generic/684 [not run] /tmp/tmp.XR9uc7m4Witest: file alloc unit 4194304 larger than op length 65536 2023-12-17T06:31:16.792 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/684 2023-12-17T06:31:16.792 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/684 2023-12-17T06:31:16.793 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:16.806 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:16.806 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/685) 2023-12-17T06:31:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:17.079 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:17.079 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:18.579 INFO:teuthology.orchestra.run.smithi028.stdout:generic/685 [not run] xfs_io fzero failed (old kernel/wrong fs?) 2023-12-17T06:31:18.587 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/685 2023-12-17T06:31:18.587 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/685 2023-12-17T06:31:18.587 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:18.602 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:18.602 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/686) 2023-12-17T06:31:18.874 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:18.881 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:18.881 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:20.374 INFO:teuthology.orchestra.run.smithi028.stdout:generic/686 [not run] xfs_io finsert failed (old kernel/wrong fs?) 2023-12-17T06:31:20.378 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/686 2023-12-17T06:31:20.378 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/686 2023-12-17T06:31:20.378 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:20.381 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:20.399 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:20.399 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/687) 2023-12-17T06:31:20.666 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:20.674 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:20.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:22.225 INFO:teuthology.orchestra.run.smithi028.stdout:generic/687 [not run] xfs_io fcollapse failed (old kernel/wrong fs?) 2023-12-17T06:31:22.229 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/687 2023-12-17T06:31:22.229 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/687 2023-12-17T06:31:22.229 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:22.232 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:22.246 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:22.246 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/688) 2023-12-17T06:31:22.478 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:22.489 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:22.489 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:22.791 INFO:teuthology.orchestra.run.smithi028.stdout:generic/688 [not run] /tmp/tmp.XR9uc7m4Witest: file alloc unit 4194304 larger than op length 65536 2023-12-17T06:31:22.796 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/688 2023-12-17T06:31:22.796 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/688 2023-12-17T06:31:22.796 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:22.799 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:22.811 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:22.811 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/689) 2023-12-17T06:31:23.045 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:23.054 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:23.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:23.247 INFO:teuthology.orchestra.run.smithi028.stdout:generic/689 [not run] vfstest not support by ceph-fuse 2023-12-17T06:31:23.251 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/689 2023-12-17T06:31:23.251 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/689 2023-12-17T06:31:23.251 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:23.254 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:23.276 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:23.277 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/690) 2023-12-17T06:31:23.523 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:23.529 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:23.529 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:23.743 INFO:teuthology.orchestra.run.smithi028.stdout:generic/690 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:23.747 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/690 2023-12-17T06:31:23.747 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/690 2023-12-17T06:31:23.747 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:23.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:23.759 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:23.760 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/691) 2023-12-17T06:31:24.031 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:24.035 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:24.035 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:24.185 INFO:teuthology.orchestra.run.smithi028.stdout:generic/691 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:24.190 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/691 2023-12-17T06:31:24.190 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/691 2023-12-17T06:31:24.191 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:24.193 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:24.203 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:24.203 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/692) 2023-12-17T06:31:24.455 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:24.460 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:24.637 INFO:teuthology.orchestra.run.smithi028.stdout:generic/692 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:24.641 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/692 2023-12-17T06:31:24.642 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/692 2023-12-17T06:31:24.642 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:24.655 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:24.655 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/693) 2023-12-17T06:31:24.964 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:24.968 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:25.184 INFO:teuthology.orchestra.run.smithi028.stdout:generic/693 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:25.188 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/693 2023-12-17T06:31:25.188 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/693 2023-12-17T06:31:25.188 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:25.205 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:25.206 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/694) 2023-12-17T06:31:25.454 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:25.458 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:26.180 INFO:teuthology.orchestra.run.smithi028.stdout:generic/694 1s 2023-12-17T06:31:26.187 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/694 2023-12-17T06:31:26.187 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:26.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:26.213 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:26.213 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/695) 2023-12-17T06:31:26.506 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:26.511 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:26.686 INFO:teuthology.orchestra.run.smithi028.stdout:generic/695 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:26.690 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/695 2023-12-17T06:31:26.691 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/695 2023-12-17T06:31:26.691 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:26.694 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:26.702 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:26.703 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/696) 2023-12-17T06:31:26.949 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:26.954 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:26.954 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:27.144 INFO:teuthology.orchestra.run.smithi028.stdout:generic/696 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:27.148 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/696 2023-12-17T06:31:27.148 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/696 2023-12-17T06:31:27.148 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:27.151 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:27.159 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:27.159 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/697) 2023-12-17T06:31:27.418 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:27.423 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:27.423 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:27.728 INFO:teuthology.orchestra.run.smithi028.stdout:generic/697 [not run] ACLs not supported by this filesystem type: ceph-fuse 2023-12-17T06:31:27.733 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/697 2023-12-17T06:31:27.733 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/697 2023-12-17T06:31:27.733 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:27.758 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:27.758 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/698) 2023-12-17T06:31:28.052 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:28.057 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:28.232 INFO:teuthology.orchestra.run.smithi028.stdout:generic/698 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:28.236 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/698 2023-12-17T06:31:28.236 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/698 2023-12-17T06:31:28.236 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:28.251 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:28.251 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/699) 2023-12-17T06:31:28.510 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:28.515 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:28.951 INFO:teuthology.orchestra.run.smithi028.stdout:generic/699 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:28.954 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/699 2023-12-17T06:31:28.955 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/699 2023-12-17T06:31:28.955 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:28.972 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:28.973 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/700) 2023-12-17T06:31:29.211 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:29.216 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:29.365 INFO:teuthology.orchestra.run.smithi028.stdout:generic/700 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:29.369 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/700 2023-12-17T06:31:29.369 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/700 2023-12-17T06:31:29.369 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:29.388 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:29.388 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/701) 2023-12-17T06:31:29.643 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:29.654 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:29.654 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:30.147 INFO:teuthology.orchestra.run.smithi028.stdout:generic/701 1s 2023-12-17T06:31:30.171 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/701 2023-12-17T06:31:30.171 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:30.174 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:30.196 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:30.197 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/702) 2023-12-17T06:31:30.839 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:30.844 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:30.844 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:31.011 INFO:teuthology.orchestra.run.smithi028.stdout:generic/702 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:31.015 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/702 2023-12-17T06:31:31.015 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/702 2023-12-17T06:31:31.015 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:31.023 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:31.040 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:31.040 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/703) 2023-12-17T06:31:31.272 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:31.276 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:31.276 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:31.477 INFO:teuthology.orchestra.run.smithi028.stdout:generic/703 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:31.481 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/703 2023-12-17T06:31:31.481 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/703 2023-12-17T06:31:31.481 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:31.484 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:31.499 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:31.499 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/704) 2023-12-17T06:31:31.752 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:31.756 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:31.756 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:32.049 INFO:teuthology.orchestra.run.smithi028.stdout:generic/704 [not run] require ceph-fuse to be valid block disk 2023-12-17T06:31:32.054 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/704 2023-12-17T06:31:32.054 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/704 2023-12-17T06:31:32.054 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:32.056 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:32.066 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:32.067 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/705) 2023-12-17T06:31:32.336 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:32.340 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:32.340 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:32.500 INFO:teuthology.orchestra.run.smithi028.stdout:generic/705 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:32.504 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/705 2023-12-17T06:31:32.504 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/705 2023-12-17T06:31:32.504 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:32.507 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:32.519 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:32.519 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/706) 2023-12-17T06:31:32.801 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:32.806 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:32.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:34.924 INFO:teuthology.orchestra.run.smithi028.stdout:generic/706 2s 2023-12-17T06:31:34.931 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/706 2023-12-17T06:31:34.931 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:34.935 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:35.079 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:35.079 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/707) 2023-12-17T06:31:35.628 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:35.633 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:35.834 INFO:teuthology.orchestra.run.smithi028.stdout:generic/707 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:31:35.841 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/707 2023-12-17T06:31:35.841 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/707 2023-12-17T06:31:35.841 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:35.845 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:35.858 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:35.858 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/708) 2023-12-17T06:31:36.091 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:36.097 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:36.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:36.766 INFO:teuthology.orchestra.run.smithi028.stdout:generic/708 0s 2023-12-17T06:31:36.772 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/708 2023-12-17T06:31:36.773 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:36.776 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:36.793 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:36.793 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/709) 2023-12-17T06:31:37.404 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:37.425 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:37.425 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:37.699 INFO:teuthology.orchestra.run.smithi028.stdout:generic/709 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/709 2023-12-17T06:31:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/709 2023-12-17T06:31:37.704 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:37.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:37.724 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:37.724 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/710) 2023-12-17T06:31:38.373 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:38.380 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:38.380 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:38.616 INFO:teuthology.orchestra.run.smithi028.stdout:generic/710 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:38.625 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/710 2023-12-17T06:31:38.625 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/710 2023-12-17T06:31:38.625 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:38.633 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:38.643 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:38.643 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/711) 2023-12-17T06:31:39.290 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:39.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:39.650 INFO:teuthology.orchestra.run.smithi028.stdout:generic/711 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/711 2023-12-17T06:31:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/711 2023-12-17T06:31:39.655 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:39.657 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:39.670 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:39.670 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/712) 2023-12-17T06:31:40.329 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:40.334 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:40.334 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:40.584 INFO:teuthology.orchestra.run.smithi028.stdout:generic/712 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:40.588 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/712 2023-12-17T06:31:40.588 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/712 2023-12-17T06:31:40.588 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:40.591 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:40.600 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:40.601 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/713) 2023-12-17T06:31:41.243 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:41.251 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:41.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:41.493 INFO:teuthology.orchestra.run.smithi028.stdout:generic/713 [not run] xfs_io swapext -v exchrange -s 64k -l 64k not supported on ceph-fuse 2023-12-17T06:31:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/713 2023-12-17T06:31:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/713 2023-12-17T06:31:41.497 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:41.500 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:41.509 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:41.510 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/714) 2023-12-17T06:31:42.152 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:42.158 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:42.392 INFO:teuthology.orchestra.run.smithi028.stdout:generic/714 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:42.397 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/714 2023-12-17T06:31:42.397 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/714 2023-12-17T06:31:42.397 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:42.415 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:42.415 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/715) 2023-12-17T06:31:43.080 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:43.085 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:43.341 INFO:teuthology.orchestra.run.smithi028.stdout:generic/715 [not run] xfs_io swapext -v exchrange -s 64k -l 64k not supported on ceph-fuse 2023-12-17T06:31:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/715 2023-12-17T06:31:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/715 2023-12-17T06:31:43.346 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:43.363 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:43.363 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/716) 2023-12-17T06:31:44.016 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:44.021 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:44.021 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:44.239 INFO:teuthology.orchestra.run.smithi028.stdout:generic/716 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:44.243 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/716 2023-12-17T06:31:44.243 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/716 2023-12-17T06:31:44.243 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:44.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:44.254 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:44.254 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/717) 2023-12-17T06:31:44.887 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:44.898 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:44.898 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:45.141 INFO:teuthology.orchestra.run.smithi028.stdout:generic/717 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:45.146 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/717 2023-12-17T06:31:45.146 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/717 2023-12-17T06:31:45.146 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:45.149 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:45.168 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:45.168 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/718) 2023-12-17T06:31:45.827 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:45.831 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:45.831 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:46.066 INFO:teuthology.orchestra.run.smithi028.stdout:generic/718 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:46.071 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/718 2023-12-17T06:31:46.071 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/718 2023-12-17T06:31:46.071 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:46.073 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:46.091 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:46.091 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/719) 2023-12-17T06:31:46.715 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:46.720 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:46.720 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:46.990 INFO:teuthology.orchestra.run.smithi028.stdout:generic/719 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:46.994 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/719 2023-12-17T06:31:46.995 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/719 2023-12-17T06:31:46.995 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:47.002 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:47.012 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:47.012 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/720) 2023-12-17T06:31:47.660 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:47.664 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:47.664 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:47.856 INFO:teuthology.orchestra.run.smithi028.stdout:generic/720 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:47.860 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/720 2023-12-17T06:31:47.860 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/720 2023-12-17T06:31:47.860 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:47.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:47.877 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:47.878 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/721) 2023-12-17T06:31:48.523 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:48.527 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:48.779 INFO:teuthology.orchestra.run.smithi028.stdout:generic/721 [not run] xfs_io startupdate support is missing 2023-12-17T06:31:48.783 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/721 2023-12-17T06:31:48.783 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/721 2023-12-17T06:31:48.783 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:48.785 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:48.794 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:48.795 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/722) 2023-12-17T06:31:49.013 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:49.018 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:49.018 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:49.260 INFO:teuthology.orchestra.run.smithi028.stdout:generic/722 [not run] xfs_io swapext -v exchrange -a not supported on ceph-fuse 2023-12-17T06:31:49.265 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/722 2023-12-17T06:31:49.265 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/722 2023-12-17T06:31:49.265 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:49.273 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:49.282 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:49.282 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/723) 2023-12-17T06:31:49.931 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:49.936 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:49.936 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:50.176 INFO:teuthology.orchestra.run.smithi028.stdout:generic/723 [not run] xfs_io swapext -v exchrange not supported on ceph-fuse 2023-12-17T06:31:50.181 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/723 2023-12-17T06:31:50.181 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/723 2023-12-17T06:31:50.181 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:50.184 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:50.192 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:50.192 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/724) 2023-12-17T06:31:50.812 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:50.823 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:50.823 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:51.064 INFO:teuthology.orchestra.run.smithi028.stdout:generic/724 [not run] xfs_io swapext -v exchrange -a not supported on ceph-fuse 2023-12-17T06:31:51.088 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/724 2023-12-17T06:31:51.088 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/724 2023-12-17T06:31:51.088 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:51.089 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:51.089 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/725) 2023-12-17T06:31:51.775 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:51.784 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:51.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:52.010 INFO:teuthology.orchestra.run.smithi028.stdout:generic/725 [not run] xfs_io swapext -v exchrange -a not supported on ceph-fuse 2023-12-17T06:31:52.026 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/725 2023-12-17T06:31:52.026 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/725 2023-12-17T06:31:52.026 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:52.039 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:52.039 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/726) 2023-12-17T06:31:52.642 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:52.647 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:52.647 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:53.211 INFO:teuthology.orchestra.run.smithi028.stdout:generic/726 [not run] xfs_io swapext -v exchrange -a not supported on ceph-fuse 2023-12-17T06:31:53.216 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/726 2023-12-17T06:31:53.216 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/726 2023-12-17T06:31:53.216 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:53.219 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:53.234 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:53.235 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/727) 2023-12-17T06:31:53.840 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:53.850 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:54.117 INFO:teuthology.orchestra.run.smithi028.stdout:generic/727 [not run] xfs_io swapext -v exchrange -a not supported on ceph-fuse 2023-12-17T06:31:54.121 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/727 2023-12-17T06:31:54.121 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/727 2023-12-17T06:31:54.121 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:54.138 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:54.138 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/728) 2023-12-17T06:31:54.784 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:54.795 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:59.064 INFO:teuthology.orchestra.run.smithi028.stdout:generic/728 5s 2023-12-17T06:31:59.070 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/728 2023-12-17T06:31:59.070 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:31:59.073 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:31:59.086 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:31:59.086 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/729) 2023-12-17T06:31:59.342 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:31:59.348 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:31:59.348 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:00.073 INFO:teuthology.orchestra.run.smithi028.stdout:generic/729 1s 2023-12-17T06:32:00.079 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/729 2023-12-17T06:32:00.080 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:32:00.083 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:00.101 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:32:00.101 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/730) 2023-12-17T06:32:00.722 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:32:00.727 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:32:00.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:00.908 INFO:teuthology.orchestra.run.smithi028.stdout:generic/730 [not run] require ceph-fuse to be valid block disk 2023-12-17T06:32:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/730 2023-12-17T06:32:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/730 2023-12-17T06:32:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:32:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:00.926 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:32:00.927 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/731) 2023-12-17T06:32:01.160 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:32:01.169 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:32:01.169 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:01.362 INFO:teuthology.orchestra.run.smithi028.stdout:generic/731 [not run] require ceph-fuse to be valid block disk 2023-12-17T06:32:01.367 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/731 2023-12-17T06:32:01.367 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/731 2023-12-17T06:32:01.367 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:32:01.375 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:01.384 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:32:01.384 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/732) 2023-12-17T06:32:01.629 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:32:01.634 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:32:01.634 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:01.792 INFO:teuthology.orchestra.run.smithi028.stdout:generic/732 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:32:01.797 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/732 2023-12-17T06:32:01.798 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/732 2023-12-17T06:32:01.798 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:32:01.801 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:01.811 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:32:01.812 DEBUG:teuthology.orchestra.run.smithi028:> (cd /tmp/tmp.nC0MjoAvrDxfstests-dev && exec sudo env DIFF_LENGTH=0 ./check generic/733) 2023-12-17T06:32:02.054 INFO:teuthology.orchestra.run.smithi028.stdout:FSTYP -- ceph-fuse 2023-12-17T06:32:02.059 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM -- Linux/x86_64 smithi028 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 2023-12-17T06:32:02.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:02.215 INFO:teuthology.orchestra.run.smithi028.stdout:generic/733 [not run] this test requires a valid $SCRATCH_DEV 2023-12-17T06:32:02.220 INFO:teuthology.orchestra.run.smithi028.stdout:Ran: generic/733 2023-12-17T06:32:02.220 INFO:teuthology.orchestra.run.smithi028.stdout:Not run: generic/733 2023-12-17T06:32:02.220 INFO:teuthology.orchestra.run.smithi028.stdout:Passed all 1 tests 2023-12-17T06:32:02.223 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:02.238 INFO:tasks.cephfs.xfstests_dev:Command return value: 0 2023-12-17T06:32:02.238 INFO:tasks.cephfs.xfstests_dev:=========================================================Total number of tests failed = 16========================================================= 2023-12-17T06:32:02.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo userdel --force --remove fsgqa 2023-12-17T06:32:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:userdel: user fsgqa is currently used by process 489510 2023-12-17T06:32:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:userdel: fsgqa mail spool (/var/mail/fsgqa) not found 2023-12-17T06:32:02.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo userdel --force --remove 123456-fsgqa 2023-12-17T06:32:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:userdel: 123456-fsgqa mail spool (/var/mail/123456-fsgqa) not found 2023-12-17T06:32:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:userdel: 123456-fsgqa home directory (/home/123456-fsgqa) not found 2023-12-17T06:32:03.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo groupdel fsgqa 2023-12-17T06:32:03.130 INFO:teuthology.orchestra.run.smithi028.stderr:groupdel: group 'fsgqa' does not exist 2023-12-17T06:32:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-12-17T06:32:03.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:03.133 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo stat /tmp/tmp.nC0MjoAvrDxfstests-dev/results) 2023-12-17T06:32:03.188 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: cd: /home/ubuntu/cephtest/mnt.0: Permission denied 2023-12-17T06:32:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T06:32:03.193 INFO:tasks.cephfs.xfstests_dev:xfstests-dev repo contains not directory named "results". repo location: /tmp/tmp.nC0MjoAvrDxfstests-dev 2023-12-17T06:32:03.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf /tmp/tmp.nC0MjoAvrDxfstests-dev 2023-12-17T06:32:03.495 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-17T06:32:03.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-17T06:32:03.640 DEBUG:teuthology.orchestra.run.smithi122:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-17T06:32:03.812 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-17T06:32:03.812 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-17T06:32:03.812 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-17T06:32:03.900 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16339660k,nr_inodes=4084915,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19728 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275312k,nr_inodes=818828,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 tmpfs /run/user/10101 tmpfs rw,nosuid,nodev,relatime,size=3275312k,nr_inodes=818828,mode=700,uid=10101,gid=10101,inode64 0 0 2023-12-17T06:32:03.900 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2023-12-17T06:32:03.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:03.900 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-17T06:32:03.908 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-17T06:32:03.908 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-17T06:32:03.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336396k,nr_inodes=4084099,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16145 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274660k,nr_inodes=818665,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-17T06:32:03.927 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2023-12-17T06:32:03.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:03.928 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-17T06:32:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T06:32:03.995 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-17T06:32:03.995 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-17T06:32:03.996 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-17T06:32:04.000 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16339660k,nr_inodes=4084915,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19728 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275312k,nr_inodes=818828,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 tmpfs /run/user/10101 tmpfs rw,nosuid,nodev,relatime,size=3275312k,nr_inodes=818828,mode=700,uid=10101,gid=10101,inode64 0 0 2023-12-17T06:32:04.000 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2023-12-17T06:32:04.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:04.001 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-17T06:32:04.046 INFO:teuthology.orchestra.run.smithi028.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-17T06:32:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T06:32:04.047 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-17T06:32:04.047 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-17T06:32:04.053 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336396k,nr_inodes=4084099,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16145 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274660k,nr_inodes=818665,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-17T06:32:04.053 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi122.front.sepia.ceph.com 2023-12-17T06:32:04.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:04.054 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-17T06:32:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T06:32:04.098 INFO:teuthology.orchestra.run.smithi122.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-17T06:32:04.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-17T06:32:04.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa080cfa640 1 Processor -- start 2023-12-17T06:32:04.791 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa080cfa640 1 -- start start 2023-12-17T06:32:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa080cfa640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 0x7fa07c0cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa080cfa640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa07c06e8a0 0x7fa07c15d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa080cfa640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c0ce840 con 0x7fa07c15cdb0 2023-12-17T06:32:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa080cfa640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa07c0ce560 con 0x7fa07c06e8a0 2023-12-17T06:32:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa080cfa640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c070f80 con 0x7fa07c159490 2023-12-17T06:32:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa073fff640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 0x7fa07c0cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa07ad9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa07c06e8a0 0x7fa07c15d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa07ad9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa07c06e8a0 0x7fa07c15d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57026/0 (socket says 172.21.15.28:57026) 2023-12-17T06:32:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa07ad9d640 1 -- 172.21.15.28:0/3372919713 learned_addr learned my addr 172.21.15.28:0/3372919713 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa07ad9d640 1 -- 172.21.15.28:0/3372919713 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 msgr2=0x7fa07c0cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa07ad9d640 1 --2- 172.21.15.28:0/3372919713 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 0x7fa07c0cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa07ad9d640 1 -- 172.21.15.28:0/3372919713 >> v1:172.21.15.122:6789/0 conn(0x7fa07c15cdb0 legacy=0x7fa07c0b1fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-17T06:32:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.787+0000 7fa07ad9d640 1 -- 172.21.15.28:0/3372919713 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c071260 con 0x7fa07c06e8a0 2023-12-17T06:32:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.791+0000 7fa07ad9d640 1 --2- 172.21.15.28:0/3372919713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c06e8a0 0x7fa07c15d190 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fa074009e50 tx=0x7fa07401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899471d93b2d4c64 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.791+0000 7fa079d9b640 1 -- 172.21.15.28:0/3372919713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0740a56b0 con 0x7fa07c06e8a0 2023-12-17T06:32:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.791+0000 7fa079d9b640 1 -- 172.21.15.28:0/3372919713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0740a5810 con 0x7fa07c06e8a0 2023-12-17T06:32:04.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.791+0000 7fa079d9b640 1 -- 172.21.15.28:0/3372919713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0740a5ba0 con 0x7fa07c06e8a0 2023-12-17T06:32:04.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.791+0000 7fa080cfa640 1 -- 172.21.15.28:0/3372919713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c06e8a0 msgr2=0x7fa07c15d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:04.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.791+0000 7fa080cfa640 1 --2- 172.21.15.28:0/3372919713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c06e8a0 0x7fa07c15d190 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fa074009e50 tx=0x7fa07401c920 comp rx=0 tx=0).stop 2023-12-17T06:32:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.795+0000 7fa080cfa640 1 -- 172.21.15.28:0/3372919713 shutdown_connections 2023-12-17T06:32:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.795+0000 7fa080cfa640 1 --2- 172.21.15.28:0/3372919713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c06e8a0 0x7fa07c15d190 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.795+0000 7fa080cfa640 1 --2- 172.21.15.28:0/3372919713 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 0x7fa07c0cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:04.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.795+0000 7fa080cfa640 1 -- 172.21.15.28:0/3372919713 >> 172.21.15.28:0/3372919713 conn(0x7fa07c06c2c0 msgr2=0x7fa07c06c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.795+0000 7fa080cfa640 1 -- 172.21.15.28:0/3372919713 shutdown_connections 2023-12-17T06:32:04.800 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.795+0000 7fa080cfa640 1 -- 172.21.15.28:0/3372919713 wait complete. 2023-12-17T06:32:04.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.807+0000 7fa080cfa640 1 Processor -- start 2023-12-17T06:32:04.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.807+0000 7fa080cfa640 1 -- start start 2023-12-17T06:32:04.815 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa080cfa640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa07c06e8a0 0x7fa07c14ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa080cfa640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 0x7fa07c151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa080cfa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c15cdb0 0x7fa07c15a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa080cfa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c163be0 con 0x7fa07c15cdb0 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa080cfa640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c162ff0 con 0x7fa07c06e8a0 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa080cfa640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c1632b0 con 0x7fa07c159490 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c15cdb0 0x7fa07c15a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07a59c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa07c06e8a0 0x7fa07c14ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c15cdb0 0x7fa07c15a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57042/0 (socket says 172.21.15.28:57042) 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 -- 172.21.15.28:0/360101794 learned_addr learned my addr 172.21.15.28:0/360101794 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa073fff640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 0x7fa07c151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 -- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 msgr2=0x7fa07c151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 0x7fa07c151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 -- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa07c06e8a0 msgr2=0x7fa07c14ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa07c06e8a0 0x7fa07c14ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 -- 172.21.15.28:0/360101794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa074009b30 con 0x7fa07c15cdb0 2023-12-17T06:32:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07ad9d640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c15cdb0 0x7fa07c15a6a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fa07401afd0 tx=0x7fa0740196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa072ffd640 1 -- 172.21.15.28:0/360101794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0740a56b0 con 0x7fa07c15cdb0 2023-12-17T06:32:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa072ffd640 1 -- 172.21.15.28:0/360101794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0740b8880 con 0x7fa07c15cdb0 2023-12-17T06:32:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07c163590 con 0x7fa07c15cdb0 2023-12-17T06:32:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa072ffd640 1 -- 172.21.15.28:0/360101794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0740b8c10 con 0x7fa07c15cdb0 2023-12-17T06:32:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07c14f9d0 con 0x7fa07c15cdb0 2023-12-17T06:32:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa072ffd640 1 -- 172.21.15.28:0/360101794 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fa0740b6440 con 0x7fa07c15cdb0 2023-12-17T06:32:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa072ffd640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa05c066470 0x7fa05c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa072ffd640 1 -- 172.21.15.28:0/360101794 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa07411f220 con 0x7fa07c15cdb0 2023-12-17T06:32:04.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.811+0000 7fa07a59c640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa05c066470 0x7fa05c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:04.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.815+0000 7fa07a59c640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa05c066470 0x7fa05c068930 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0098e0 tx=0x7fa06c009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:04.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.815+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07c1620b0 con 0x7fa07c15cdb0 2023-12-17T06:32:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:04.827+0000 7fa072ffd640 1 -- 172.21.15.28:0/360101794 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fa0740ec760 con 0x7fa07c15cdb0 2023-12-17T06:32:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.119+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa07c069440 con 0x7fa07c15cdb0 2023-12-17T06:32:05.129 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.123+0000 7fa072ffd640 1 -- 172.21.15.28:0/360101794 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fa0740bc020 con 0x7fa07c15cdb0 2023-12-17T06:32:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:05.130 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/992095456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":992095456},{"type":"v1","addr":"172.21.15.122:6835","nonce":992095456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3174619236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3174619236},{"type":"v1","addr":"172.21.15.122:6837","nonce":3174619236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4691,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3342817962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3342817962},{"type":"v1","addr":"172.21.15.28:6835","nonce":3342817962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:58:40.841493+0000","modified":"2023-12-16T21:58:42.871930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5193},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5193":{"gid":5193,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/924398795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":924398795},{"type":"v1","addr":"172.21.15.28:6833","nonce":924398795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-17T06:32:05.130 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 15 2023-12-17T06:32:05.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.127+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa05c066470 msgr2=0x7fa05c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.127+0000 7fa080cfa640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa05c066470 0x7fa05c068930 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0098e0 tx=0x7fa06c009060 comp rx=0 tx=0).stop 2023-12-17T06:32:05.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.127+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c15cdb0 msgr2=0x7fa07c15a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.127+0000 7fa080cfa640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c15cdb0 0x7fa07c15a6a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fa07401afd0 tx=0x7fa0740196c0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.131+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 shutdown_connections 2023-12-17T06:32:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.131+0000 7fa080cfa640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa05c066470 0x7fa05c068930 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.131+0000 7fa080cfa640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07c15cdb0 0x7fa07c15a6a0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.131+0000 7fa080cfa640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa07c159490 0x7fa07c151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.131+0000 7fa080cfa640 1 --2- 172.21.15.28:0/360101794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa07c06e8a0 0x7fa07c14ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.131+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 >> 172.21.15.28:0/360101794 conn(0x7fa07c06c2c0 msgr2=0x7fa07c0cb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.131+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 shutdown_connections 2023-12-17T06:32:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.131+0000 7fa080cfa640 1 -- 172.21.15.28:0/360101794 wait complete. 2023-12-17T06:32:05.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-17T06:32:05.335 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.327+0000 7f4433a4c640 1 Processor -- start 2023-12-17T06:32:05.335 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.327+0000 7f4433a4c640 1 -- start start 2023-12-17T06:32:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4433a4c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c0cfe90 0x7f442c0d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4433a4c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c0d3880 con 0x7f442c0b3c70 2023-12-17T06:32:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4433a4c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f442c0cdc80 con 0x7f442c0b35b0 2023-12-17T06:32:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4433a4c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c0d4090 con 0x7f442c0cfe90 2023-12-17T06:32:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4431fe9640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f442c0b3c70 0x7f442c0ce440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40340/0 (socket says 172.21.15.28:40340) 2023-12-17T06:32:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4431fe9640 1 -- 172.21.15.28:0/4262549194 learned_addr learned my addr 172.21.15.28:0/4262549194 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860307461 0 0) 0x7f442c0d3880 con 0x7f442c0b3c70 2023-12-17T06:32:05.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4418003530 con 0x7f442c0b3c70 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908444961 0 0) 0x7f442c0cdc80 con 0x7f442c0b35b0 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f442c0d3880 con 0x7f442c0b35b0 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3876610316 0 0) 0x7f4418003530 con 0x7f442c0b3c70 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c0cfe90 msgr2=0x7f442c0d0f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c0cfe90 0x7f442c0d0f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 >> v1:172.21.15.122:6789/0 conn(0x7f442c0b35b0 legacy=0x7f442c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c0d3ad0 con 0x7f442c0b3c70 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f4424001a00 con 0x7f442c0b3c70 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 >> v1:172.21.15.28:6789/0 conn(0x7f442c0b3c70 legacy=0x7f442c0ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4418003b40 0x7f4418003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44180047d0 0x7f4418004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f44180052b0 0x7f441800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c0d4090 con 0x7f44180047d0 2023-12-17T06:32:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441800c160 con 0x7f4418003b40 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441800c320 con 0x7f44180052b0 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f44317e8640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4418003b40 0x7f4418003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4431fe9640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f44180052b0 0x7f441800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4431fe9640 1 -- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4418003b40 msgr2=0x7f4418003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4431fe9640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4418003b40 0x7f4418003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4431fe9640 1 -- 172.21.15.28:0/4262549194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44180047d0 msgr2=0x7f4418004c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4431fe9640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44180047d0 0x7f4418004c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f4431fe9640 1 -- 172.21.15.28:0/4262549194 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c0d3ad0 con 0x7f44180052b0 2023-12-17T06:32:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.331+0000 7f44317e8640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4418003b40 0x7f4418003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.335+0000 7f4431fe9640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f44180052b0 0x7f441800b990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f442400a930 tx=0x7f442401d370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5fdf32a8edad83f server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:05.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.335+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44240ad630 con 0x7f44180052b0 2023-12-17T06:32:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.343+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44240a8070 con 0x7f44180052b0 2023-12-17T06:32:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.343+0000 7f4423fff640 1 -- 172.21.15.28:0/4262549194 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44240ad9d0 con 0x7f44180052b0 2023-12-17T06:32:05.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.343+0000 7f4433a4c640 1 -- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f44180052b0 msgr2=0x7f441800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.343+0000 7f4433a4c640 1 --2- 172.21.15.28:0/4262549194 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f44180052b0 0x7f441800b990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f442400a930 tx=0x7f442401d370 comp rx=0 tx=0).stop 2023-12-17T06:32:05.349 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.343+0000 7f4431fe9640 1 -- 172.21.15.28:0/4262549194 reap_dead start 2023-12-17T06:32:05.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.347+0000 7f4433a4c640 1 -- 172.21.15.28:0/4262549194 shutdown_connections 2023-12-17T06:32:05.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.347+0000 7f4433a4c640 1 -- 172.21.15.28:0/4262549194 >> 172.21.15.28:0/4262549194 conn(0x7f442c09c4b0 msgr2=0x7f442c0d3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:05.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.347+0000 7f4433a4c640 1 -- 172.21.15.28:0/4262549194 shutdown_connections 2023-12-17T06:32:05.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.351+0000 7f4433a4c640 1 -- 172.21.15.28:0/4262549194 wait complete. 2023-12-17T06:32:05.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 Processor -- start 2023-12-17T06:32:05.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 -- start start 2023-12-17T06:32:05.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44180052b0 0x7f442c162f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f442c1f97e0 0x7f442c0cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c1f9c60 0x7f442c0ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c06d9f0 con 0x7f44180052b0 2023-12-17T06:32:05.361 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c162a80 con 0x7f442c1f97e0 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442c161c60 con 0x7f442c1f9c60 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c1f9c60 0x7f442c0ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c1f9c60 0x7f442c0ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:49428/0 (socket says 172.21.15.28:49428) 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 -- 172.21.15.28:0/1799018415 learned_addr learned my addr 172.21.15.28:0/1799018415 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 -- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f442c1f97e0 msgr2=0x7f442c0cc850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f442c1f97e0 0x7f442c0cc850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 -- 172.21.15.28:0/1799018415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44180052b0 msgr2=0x7f442c162f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44180052b0 0x7f442c162f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.362 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 -- 172.21.15.28:0/1799018415 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c162150 con 0x7f442c1f9c60 2023-12-17T06:32:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4431fe9640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c1f9c60 0x7f442c0ccd90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f442401d1a0 tx=0x7f4418005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4421ffb640 1 -- 172.21.15.28:0/1799018415 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424001800 con 0x7f442c1f9c60 2023-12-17T06:32:05.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4421ffb640 1 -- 172.21.15.28:0/1799018415 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4424001960 con 0x7f442c1f9c60 2023-12-17T06:32:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.355+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442c162430 con 0x7f442c1f9c60 2023-12-17T06:32:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.359+0000 7f4421ffb640 1 -- 172.21.15.28:0/1799018415 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424004d70 con 0x7f442c1f9c60 2023-12-17T06:32:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.359+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c160f30 con 0x7f442c1f9c60 2023-12-17T06:32:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.359+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442c069440 con 0x7f442c1f9c60 2023-12-17T06:32:05.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.359+0000 7f4421ffb640 1 -- 172.21.15.28:0/1799018415 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f44240aa030 con 0x7f442c1f9c60 2023-12-17T06:32:05.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.359+0000 7f4421ffb640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f44100664c0 0x7f4410068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.365 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.359+0000 7f4421ffb640 1 -- 172.21.15.28:0/1799018415 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f442411da10 con 0x7f442c1f9c60 2023-12-17T06:32:05.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.359+0000 7f44317e8640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f44100664c0 0x7f4410068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.367+0000 7f4421ffb640 1 -- 172.21.15.28:0/1799018415 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f44240eb020 con 0x7f442c1f9c60 2023-12-17T06:32:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.367+0000 7f44317e8640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f44100664c0 0x7f4410068980 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f442c071270 tx=0x7f4428009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.603+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f442c162bc0 con 0x7f442c1f9c60 2023-12-17T06:32:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.627+0000 7f4421ffb640 1 -- 172.21.15.28:0/1799018415 <== mon.2 v2:172.21.15.122:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f442c162bc0 con 0x7f442c1f9c60 2023-12-17T06:32:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:05.635 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:42.498586+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:58:38.932657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:58:39.841920+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-17T06:32:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f44100664c0 msgr2=0x7f4410068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f44100664c0 0x7f4410068980 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f442c071270 tx=0x7f4428009040 comp rx=0 tx=0).stop 2023-12-17T06:32:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c1f9c60 msgr2=0x7f442c0ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c1f9c60 0x7f442c0ccd90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f442401d1a0 tx=0x7f4418005800 comp rx=0 tx=0).stop 2023-12-17T06:32:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 shutdown_connections 2023-12-17T06:32:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f44100664c0 0x7f4410068980 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f442c1f9c60 0x7f442c0ccd90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f442c1f97e0 0x7f442c0cc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 --2- 172.21.15.28:0/1799018415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44180052b0 0x7f442c162f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 >> 172.21.15.28:0/1799018415 conn(0x7f442c09c4b0 msgr2=0x7f442c06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:05.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 shutdown_connections 2023-12-17T06:32:05.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.631+0000 7f4433a4c640 1 -- 172.21.15.28:0/1799018415 wait complete. 2023-12-17T06:32:05.659 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:57:31.458376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:58:38.932657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:58:39.841920+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-17T06:32:05.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-17T06:32:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.791+0000 7f83c39c4640 1 Processor -- start 2023-12-17T06:32:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c39c4640 1 -- start start 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c39c4640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83bc15adb0 0x7f83bc13d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c39c4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc13f5f0 0x7f83bc0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c39c4640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc13d2b0 con 0x7f83bc071d90 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c39c4640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83bc140520 con 0x7f83bc15adb0 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c39c4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0cd7f0 con 0x7f83bc13f5f0 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c0f5f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83bc15adb0 0x7f83bc13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c0f5f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83bc15adb0 0x7f83bc13d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57066/0 (socket says 172.21.15.28:57066) 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c0f5f640 1 -- 172.21.15.28:0/792709452 learned_addr learned my addr 172.21.15.28:0/792709452 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c1f61640 1 --2- 172.21.15.28:0/792709452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc13f5f0 0x7f83bc0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c0f5f640 1 -- 172.21.15.28:0/792709452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc13f5f0 msgr2=0x7f83bc0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c0f5f640 1 --2- 172.21.15.28:0/792709452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc13f5f0 0x7f83bc0ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c0f5f640 1 -- 172.21.15.28:0/792709452 >> v1:172.21.15.28:6789/0 conn(0x7f83bc071d90 legacy=0x7f83bc072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c0f5f640 1 -- 172.21.15.28:0/792709452 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc0cdd70 con 0x7f83bc15adb0 2023-12-17T06:32:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.795+0000 7f83c1f61640 1 --2- 172.21.15.28:0/792709452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc13f5f0 0x7f83bc0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-17T06:32:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.799+0000 7f83c0f5f640 1 --2- 172.21.15.28:0/792709452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc13d560 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009870 tx=0x7f83b801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a5b77330c94235 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83b3fff640 1 -- 172.21.15.28:0/792709452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b80aa5f0 con 0x7f83bc15adb0 2023-12-17T06:32:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83b3fff640 1 -- 172.21.15.28:0/792709452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b80aa750 con 0x7f83bc15adb0 2023-12-17T06:32:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83b3fff640 1 -- 172.21.15.28:0/792709452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b80aaae0 con 0x7f83bc15adb0 2023-12-17T06:32:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- 172.21.15.28:0/792709452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 msgr2=0x7f83bc13d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 --2- 172.21.15.28:0/792709452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc13d560 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009870 tx=0x7f83b801c4d0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- 172.21.15.28:0/792709452 shutdown_connections 2023-12-17T06:32:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 --2- 172.21.15.28:0/792709452 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc13f5f0 0x7f83bc0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 --2- 172.21.15.28:0/792709452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc13d560 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- 172.21.15.28:0/792709452 >> 172.21.15.28:0/792709452 conn(0x7f83bc0b0cc0 msgr2=0x7f83bc0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- 172.21.15.28:0/792709452 shutdown_connections 2023-12-17T06:32:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- 172.21.15.28:0/792709452 wait complete. 2023-12-17T06:32:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 Processor -- start 2023-12-17T06:32:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- start start 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc071d90 0x7f83bc0b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc13f5f0 0x7f83bc13bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b22f0 con 0x7f83bc15adb0 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b2570 con 0x7f83bc13f5f0 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c39c4640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0b2850 con 0x7f83bc071d90 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1f61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1f61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57070/0 (socket says 172.21.15.28:57070) 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1f61640 1 -- 172.21.15.28:0/1232332819 learned_addr learned my addr 172.21.15.28:0/1232332819 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1760640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc071d90 0x7f83bc0b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c0f5f640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc13f5f0 0x7f83bc13bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1f61640 1 -- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc071d90 msgr2=0x7f83bc0b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1f61640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc071d90 0x7f83bc0b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1f61640 1 -- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc13f5f0 msgr2=0x7f83bc13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:05.812 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1f61640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc13f5f0 0x7f83bc13bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:05.812 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.803+0000 7f83c1f61640 1 -- 172.21.15.28:0/1232332819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b80094d0 con 0x7f83bc15adb0 2023-12-17T06:32:05.812 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83c1f61640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc158f40 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009fa0 tx=0x7f83b400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83b1ffb640 1 -- 172.21.15.28:0/1232332819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4003ba0 con 0x7f83bc15adb0 2023-12-17T06:32:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83b1ffb640 1 -- 172.21.15.28:0/1232332819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b400f040 con 0x7f83bc15adb0 2023-12-17T06:32:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc0b2b30 con 0x7f83bc15adb0 2023-12-17T06:32:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc0b30f0 con 0x7f83bc15adb0 2023-12-17T06:32:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83b1ffb640 1 -- 172.21.15.28:0/1232332819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4004030 con 0x7f83bc15adb0 2023-12-17T06:32:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83b1ffb640 1 -- 172.21.15.28:0/1232332819 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f83b4004250 con 0x7f83bc15adb0 2023-12-17T06:32:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc0b3330 con 0x7f83bc15adb0 2023-12-17T06:32:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83b1ffb640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83a00664a0 0x7f83a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83b1ffb640 1 -- 172.21.15.28:0/1232332819 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83b4084090 con 0x7f83bc15adb0 2023-12-17T06:32:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83c1760640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83a00664a0 0x7f83a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.807+0000 7f83c1760640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83a00664a0 0x7f83a0068960 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0098e0 tx=0x7f83ac009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:05.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:05.815+0000 7f83b1ffb640 1 -- 172.21.15.28:0/1232332819 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f83bc0b3330 con 0x7f83bc15adb0 2023-12-17T06:32:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.047+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83bc0b3330 con 0x7f83bc15adb0 2023-12-17T06:32:06.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.047+0000 7f83b1ffb640 1 -- 172.21.15.28:0/1232332819 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83bc0b3330 con 0x7f83bc15adb0 2023-12-17T06:32:06.054 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83a00664a0 msgr2=0x7f83a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83a00664a0 0x7f83a0068960 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0098e0 tx=0x7f83ac009060 comp rx=0 tx=0).stop 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 msgr2=0x7f83bc158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc158f40 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009fa0 tx=0x7f83b400d560 comp rx=0 tx=0).stop 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 shutdown_connections 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f83a00664a0 0x7f83a0068960 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc15adb0 0x7f83bc158f40 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83bc13f5f0 0x7f83bc13bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 --2- 172.21.15.28:0/1232332819 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f83bc071d90 0x7f83bc0b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.051+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 >> 172.21.15.28:0/1232332819 conn(0x7f83bc0b0cc0 msgr2=0x7f83bc13dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.055+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 shutdown_connections 2023-12-17T06:32:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.055+0000 7f83c39c4640 1 -- 172.21.15.28:0/1232332819 wait complete. 2023-12-17T06:32:06.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-17T06:32:06.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.230+0000 7f4ded1cd640 1 Processor -- start 2023-12-17T06:32:06.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4ded1cd640 1 -- start start 2023-12-17T06:32:06.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4ded1cd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f4de8143c70 0x7f4de806af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4ded1cd640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de8144810 con 0x7f4de80b25f0 2023-12-17T06:32:06.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4ded1cd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de815f2a0 con 0x7f4de8144050 2023-12-17T06:32:06.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4ded1cd640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de815f580 con 0x7f4de8143c70 2023-12-17T06:32:06.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4de759e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4de80b25f0 0x7f4de80cef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40358/0 (socket says 172.21.15.28:40358) 2023-12-17T06:32:06.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4de759e640 1 -- 172.21.15.28:0/3526474412 learned_addr learned my addr 172.21.15.28:0/3526474412 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:06.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4de6d9d640 1 --2- 172.21.15.28:0/3526474412 >> v2:172.21.15.122:3300/0 conn(0x7f4de8143c70 0x7f4de806af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:06.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4de6d9d640 1 -- 172.21.15.28:0/3526474412 >> v1:172.21.15.122:6789/0 conn(0x7f4de8144050 legacy=0x7f4de80b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.234+0000 7f4de6d9d640 1 -- 172.21.15.28:0/3526474412 >> v1:172.21.15.28:6789/0 conn(0x7f4de80b25f0 legacy=0x7f4de80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.238+0000 7f4de5d9b640 1 -- 172.21.15.28:0/3526474412 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390642727 0 0) 0x7f4de8144810 con 0x7f4de80b25f0 2023-12-17T06:32:06.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.238+0000 7f4de6d9d640 1 -- 172.21.15.28:0/3526474412 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de815f860 con 0x7f4de8143c70 2023-12-17T06:32:06.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.238+0000 7f4de6d9d640 1 --2- 172.21.15.28:0/3526474412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 0x7f4de806af40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0097c0 tx=0x7f4ddc01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24c25df5164430d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:06.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.238+0000 7f4de5d9b640 1 -- 172.21.15.28:0/3526474412 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc0ab670 con 0x7f4de8143c70 2023-12-17T06:32:06.246 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.242+0000 7f4de5d9b640 1 -- 172.21.15.28:0/3526474412 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc0ab7d0 con 0x7f4de8143c70 2023-12-17T06:32:06.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.242+0000 7f4de5d9b640 1 -- 172.21.15.28:0/3526474412 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc0abb60 con 0x7f4de8143c70 2023-12-17T06:32:06.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.242+0000 7f4ded1cd640 1 -- 172.21.15.28:0/3526474412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 msgr2=0x7f4de806af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.242+0000 7f4ded1cd640 1 --2- 172.21.15.28:0/3526474412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 0x7f4de806af40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0097c0 tx=0x7f4ddc01c450 comp rx=0 tx=0).stop 2023-12-17T06:32:06.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.250+0000 7f4ded1cd640 1 -- 172.21.15.28:0/3526474412 shutdown_connections 2023-12-17T06:32:06.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.250+0000 7f4ded1cd640 1 --2- 172.21.15.28:0/3526474412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 0x7f4de806af40 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.256 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.250+0000 7f4ded1cd640 1 -- 172.21.15.28:0/3526474412 >> 172.21.15.28:0/3526474412 conn(0x7f4de80b54d0 msgr2=0x7f4de80b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.250+0000 7f4ded1cd640 1 -- 172.21.15.28:0/3526474412 shutdown_connections 2023-12-17T06:32:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.250+0000 7f4ded1cd640 1 -- 172.21.15.28:0/3526474412 wait complete. 2023-12-17T06:32:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.250+0000 7f4ded1cd640 1 Processor -- start 2023-12-17T06:32:06.258 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.250+0000 7f4ded1cd640 1 -- start start 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4de80b25f0 0x7f4de813f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 0x7f4de80c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de8144050 0x7f4de80c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de813fa70 con 0x7f4de8144050 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80c5c50 con 0x7f4de8143c70 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80c5f30 con 0x7f4de80b25f0 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de6d9d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4de80b25f0 0x7f4de813f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de659c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 0x7f4de80c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de6d9d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4de80b25f0 0x7f4de813f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:49446/0 (socket says 172.21.15.28:49446) 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de6d9d640 1 -- 172.21.15.28:0/2056689157 learned_addr learned my addr 172.21.15.28:0/2056689157 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de659c640 1 -- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4de80b25f0 msgr2=0x7f4de813f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de659c640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4de80b25f0 0x7f4de813f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de659c640 1 -- 172.21.15.28:0/2056689157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de8144050 msgr2=0x7f4de80c7cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-17T06:32:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de659c640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de8144050 0x7f4de80c7cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de659c640 1 -- 172.21.15.28:0/2056689157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc009420 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de6d9d640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4de80b25f0 0x7f4de813f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4de659c640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 0x7f4de80c77b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4de81444e0 tx=0x7f4dd800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4dcf7fe640 1 -- 172.21.15.28:0/2056689157 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8005220 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4dcf7fe640 1 -- 172.21.15.28:0/2056689157 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd8005380 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4dcf7fe640 1 -- 172.21.15.28:0/2056689157 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8007230 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de80c6210 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de80c67d0 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc8005260 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4dcf7fe640 1 -- 172.21.15.28:0/2056689157 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4dd8007390 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.254+0000 7f4dcf7fe640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4dd4066470 0x7f4dd4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.258+0000 7f4dcf7fe640 1 -- 172.21.15.28:0/2056689157 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4dd8083230 con 0x7f4de8143c70 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.258+0000 7f4de6d9d640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4dd4066470 0x7f4dd4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.258+0000 7f4de6d9d640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4dd4066470 0x7f4dd4068930 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc01c490 tx=0x7f4ddc0a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.262+0000 7f4dcf7fe640 1 -- 172.21.15.28:0/2056689157 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f4dd80506a0 con 0x7f4de8143c70 2023-12-17T06:32:06.506 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.498+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4dc8005a90 con 0x7f4de8143c70 2023-12-17T06:32:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.502+0000 7f4dcf7fe640 1 -- 172.21.15.28:0/2056689157 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4dd8053cc0 con 0x7f4de8143c70 2023-12-17T06:32:06.507 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2023-12-17T06:32:06.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.502+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4dd4066470 msgr2=0x7f4dd4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.502+0000 7f4ded1cd640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4dd4066470 0x7f4dd4068930 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc01c490 tx=0x7f4ddc0a7000 comp rx=0 tx=0).stop 2023-12-17T06:32:06.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.502+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 msgr2=0x7f4de80c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.502+0000 7f4ded1cd640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 0x7f4de80c77b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4de81444e0 tx=0x7f4dd800c820 comp rx=0 tx=0).stop 2023-12-17T06:32:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.506+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 shutdown_connections 2023-12-17T06:32:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.506+0000 7f4ded1cd640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4dd4066470 0x7f4dd4068930 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.506+0000 7f4ded1cd640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de8144050 0x7f4de80c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.506+0000 7f4ded1cd640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de8143c70 0x7f4de80c77b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.506+0000 7f4ded1cd640 1 --2- 172.21.15.28:0/2056689157 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f4de80b25f0 0x7f4de813f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.506+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 >> 172.21.15.28:0/2056689157 conn(0x7f4de80b54d0 msgr2=0x7f4de80b58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.506+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 shutdown_connections 2023-12-17T06:32:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.506+0000 7f4ded1cd640 1 -- 172.21.15.28:0/2056689157 wait complete. 2023-12-17T06:32:06.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-17T06:32:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.686+0000 7f0946c52640 1 Processor -- start 2023-12-17T06:32:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.686+0000 7f0946c52640 1 -- start start 2023-12-17T06:32:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.686+0000 7f0946c52640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09400cb560 0x7f0940165c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.686+0000 7f0946c52640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09400cf2d0 con 0x7f094006ef40 2023-12-17T06:32:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.686+0000 7f0946c52640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0940166190 con 0x7f09401616d0 2023-12-17T06:32:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.686+0000 7f0946c52640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09400d0d80 con 0x7f09400cb560 2023-12-17T06:32:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f09449ee640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f09401616d0 0x7f094006e830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:57572/0 (socket says 172.21.15.28:57572) 2023-12-17T06:32:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f09449ee640 1 -- 172.21.15.28:0/2581673185 learned_addr learned my addr 172.21.15.28:0/2581673185 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621310034 0 0) 0x7f0930001120 con 0x7f09401616d0 2023-12-17T06:32:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0928003530 con 0x7f09401616d0 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555933560 0 0) 0x7f0928003530 con 0x7f09401616d0 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09400cb560 msgr2=0x7f0940165c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09400cb560 0x7f0940165c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 >> v1:172.21.15.28:6789/0 conn(0x7f094006ef40 legacy=0x7f094006b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09400cba10 con 0x7f09401616d0 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f0930001810 con 0x7f09401616d0 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 >> v1:172.21.15.122:6789/0 conn(0x7f09401616d0 legacy=0x7f094006e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0928003980 0x7f0928003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09280047d0 0x7f0928004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09280051e0 0x7f092800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09400d0d80 con 0x7f09280047d0 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092800c0c0 con 0x7f09280051e0 2023-12-17T06:32:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092800c280 con 0x7f0928003980 2023-12-17T06:32:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093ffff640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09280047d0 0x7f0928004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093ffff640 1 -- 172.21.15.28:0/2581673185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0928003980 msgr2=0x7f0928003d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-17T06:32:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093ffff640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0928003980 0x7f0928003d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093ffff640 1 -- 172.21.15.28:0/2581673185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09280051e0 msgr2=0x7f092800b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-17T06:32:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093ffff640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09280051e0 0x7f092800b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.690+0000 7f093ffff640 1 -- 172.21.15.28:0/2581673185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0928003530 con 0x7f09280047d0 2023-12-17T06:32:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f093ffff640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09280047d0 0x7f0928004c50 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f0934001650 tx=0x7f093401c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e3b99de602c3 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934005270 con 0x7f09280047d0 2023-12-17T06:32:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09340a7070 con 0x7f09280047d0 2023-12-17T06:32:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f093f7fe640 1 -- 172.21.15.28:0/2581673185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934003860 con 0x7f09280047d0 2023-12-17T06:32:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f0946c52640 1 -- 172.21.15.28:0/2581673185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09280047d0 msgr2=0x7f0928004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f0946c52640 1 --2- 172.21.15.28:0/2581673185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09280047d0 0x7f0928004c50 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f0934001650 tx=0x7f093401c300 comp rx=0 tx=0).stop 2023-12-17T06:32:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f09451ef640 1 -- 172.21.15.28:0/2581673185 reap_dead start 2023-12-17T06:32:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f0946c52640 1 -- 172.21.15.28:0/2581673185 shutdown_connections 2023-12-17T06:32:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.694+0000 7f0946c52640 1 -- 172.21.15.28:0/2581673185 >> 172.21.15.28:0/2581673185 conn(0x7f09400d0f10 msgr2=0x7f09400d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.698+0000 7f0946c52640 1 -- 172.21.15.28:0/2581673185 shutdown_connections 2023-12-17T06:32:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.698+0000 7f0946c52640 1 -- 172.21.15.28:0/2581673185 wait complete. 2023-12-17T06:32:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 Processor -- start 2023-12-17T06:32:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 -- start start 2023-12-17T06:32:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09280047d0 0x7f0940141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.710 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0940156e40 0x7f09400b5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09401572c0 0x7f09400b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0940156e40 0x7f09400b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0940156e40 0x7f09400b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57084/0 (socket says 172.21.15.28:57084) 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 -- 172.21.15.28:0/1024183590 learned_addr learned my addr 172.21.15.28:0/1024183590 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0940158ec0 con 0x7f0940156e40 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09401591a0 con 0x7f09401572c0 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0940159480 con 0x7f09280047d0 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f09451ef640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09401572c0 0x7f09400b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 -- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09280047d0 msgr2=0x7f0940141a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09280047d0 0x7f0940141a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 -- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09401572c0 msgr2=0x7f09400b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09401572c0 0x7f09400b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 -- 172.21.15.28:0/1024183590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0940159760 con 0x7f0940156e40 2023-12-17T06:32:06.711 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093ffff640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0940156e40 0x7f09400b5270 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f0934001650 tx=0x7f0928005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093d7fa640 1 -- 172.21.15.28:0/1024183590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09340b3070 con 0x7f0940156e40 2023-12-17T06:32:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093d7fa640 1 -- 172.21.15.28:0/1024183590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09340070f0 con 0x7f0940156e40 2023-12-17T06:32:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f093d7fa640 1 -- 172.21.15.28:0/1024183590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934007480 con 0x7f0940156e40 2023-12-17T06:32:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0940158ae0 con 0x7f0940156e40 2023-12-17T06:32:06.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.702+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0940156b30 con 0x7f0940156e40 2023-12-17T06:32:06.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.706+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09400674d0 con 0x7f0940156e40 2023-12-17T06:32:06.715 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.710+0000 7f093d7fa640 1 -- 172.21.15.28:0/1024183590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f09340a9020 con 0x7f0940156e40 2023-12-17T06:32:06.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.710+0000 7f093d7fa640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f092c066470 0x7f092c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:06.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.710+0000 7f093d7fa640 1 -- 172.21.15.28:0/1024183590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f093411cb70 con 0x7f0940156e40 2023-12-17T06:32:06.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.710+0000 7f09449ee640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f092c066470 0x7f092c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.710+0000 7f09449ee640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f092c066470 0x7f092c068930 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0930004880 tx=0x7f0930001120 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:06.720 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.714+0000 7f093d7fa640 1 -- 172.21.15.28:0/1024183590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f09340e9e40 con 0x7f0940156e40 2023-12-17T06:32:06.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.954+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f094006cdd0 con 0x7f0940156e40 2023-12-17T06:32:06.959 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.954+0000 7f093d7fa640 1 -- 172.21.15.28:0/1024183590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09340ed460 con 0x7f0940156e40 2023-12-17T06:32:06.963 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2023-12-17T06:32:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f092c066470 msgr2=0x7f092c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f092c066470 0x7f092c068930 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0930004880 tx=0x7f0930001120 comp rx=0 tx=0).stop 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0940156e40 msgr2=0x7f09400b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0940156e40 0x7f09400b5270 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f0934001650 tx=0x7f0928005800 comp rx=0 tx=0).stop 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 shutdown_connections 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f092c066470 0x7f092c068930 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f09401572c0 0x7f09400b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0940156e40 0x7f09400b5270 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 --2- 172.21.15.28:0/1024183590 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f09280047d0 0x7f0940141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.966+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 >> 172.21.15.28:0/1024183590 conn(0x7f09400d0f10 msgr2=0x7f09400b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:06.975 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.970+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 shutdown_connections 2023-12-17T06:32:06.976 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:06.970+0000 7f0946c52640 1 -- 172.21.15.28:0/1024183590 wait complete. 2023-12-17T06:32:07.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-17T06:32:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.146+0000 7f3ca44db640 1 Processor -- start 2023-12-17T06:32:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.146+0000 7f3ca44db640 1 -- start start 2023-12-17T06:32:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca44db640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 0x7f3c9c15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca44db640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c070460 con 0x7f3c9c150960 2023-12-17T06:32:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca44db640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c15ab20 con 0x7f3c9c06cdb0 2023-12-17T06:32:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca44db640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c070690 con 0x7f3c9c06d190 2023-12-17T06:32:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1a76640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3c9c150960 0x7f3c9c159da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40386/0 (socket says 172.21.15.28:40386) 2023-12-17T06:32:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1a76640 1 -- 172.21.15.28:0/146898995 learned_addr learned my addr 172.21.15.28:0/146898995 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca2a78640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 0x7f3c9c15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734777195 0 0) 0x7f3c9c070460 con 0x7f3c9c150960 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c88003530 con 0x7f3c9c150960 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980317035 0 0) 0x7f3c9c15ab20 con 0x7f3c9c06cdb0 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c9c070460 con 0x7f3c9c06cdb0 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2177095872 0 0) 0x7f3c88003530 con 0x7f3c9c150960 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 msgr2=0x7f3c9c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 0x7f3c9c15a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 >> v1:172.21.15.122:6789/0 conn(0x7f3c9c06cdb0 legacy=0x7f3c9c150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c070970 con 0x7f3c9c150960 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f3c8c001680 con 0x7f3c9c150960 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 >> v1:172.21.15.28:6789/0 conn(0x7f3c9c150960 legacy=0x7f3c9c159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c88003b40 0x7f3c88003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c880047d0 0x7f3c88004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 0x7f3c88005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c070690 con 0x7f3c880047d0 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8800ba40 con 0x7f3c88003b40 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8800bc00 con 0x7f3c9c06d190 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca2277640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c88003b40 0x7f3c88003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca2a78640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 0x7f3c88005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca1a76640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c880047d0 0x7f3c88004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca2a78640 1 -- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c88003b40 msgr2=0x7f3c88003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca2a78640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c88003b40 0x7f3c88003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca2a78640 1 -- 172.21.15.28:0/146898995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c880047d0 msgr2=0x7f3c88004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca2a78640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c880047d0 0x7f3c88004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.150+0000 7f3ca2a78640 1 -- 172.21.15.28:0/146898995 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c070970 con 0x7f3c9c06d190 2023-12-17T06:32:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca2a78640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 0x7f3c88005190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009fc0 tx=0x7f3c9401cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe202f4ced85ea10 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c940a5900 con 0x7f3c9c06d190 2023-12-17T06:32:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9400f040 con 0x7f3c9c06d190 2023-12-17T06:32:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca1275640 1 -- 172.21.15.28:0/146898995 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c940a5ca0 con 0x7f3c9c06d190 2023-12-17T06:32:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca44db640 1 -- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 msgr2=0x7f3c88005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca44db640 1 --2- 172.21.15.28:0/146898995 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c06d190 0x7f3c88005190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009fc0 tx=0x7f3c9401cca0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca2a78640 1 -- 172.21.15.28:0/146898995 reap_dead start 2023-12-17T06:32:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca44db640 1 -- 172.21.15.28:0/146898995 shutdown_connections 2023-12-17T06:32:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca44db640 1 -- 172.21.15.28:0/146898995 >> 172.21.15.28:0/146898995 conn(0x7f3c9c06a530 msgr2=0x7f3c9c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca44db640 1 -- 172.21.15.28:0/146898995 shutdown_connections 2023-12-17T06:32:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.154+0000 7f3ca44db640 1 -- 172.21.15.28:0/146898995 wait complete. 2023-12-17T06:32:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 Processor -- start 2023-12-17T06:32:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 -- start start 2023-12-17T06:32:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c14c580 0x7f3c9c0b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c14c960 0x7f3c9c14af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0b09a0 0x7f3c9c0b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c0b0510 con 0x7f3c9c14c960 2023-12-17T06:32:07.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c0b2ba0 con 0x7f3c9c0b09a0 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c14b480 con 0x7f3c9c14c580 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c14c960 0x7f3c9c14af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c14c960 0x7f3c9c14af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57108/0 (socket says 172.21.15.28:57108) 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 -- 172.21.15.28:0/3632460549 learned_addr learned my addr 172.21.15.28:0/3632460549 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca2a78640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0b09a0 0x7f3c9c0b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 -- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c14c580 msgr2=0x7f3c9c0b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c14c580 0x7f3c9c0b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 -- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0b09a0 msgr2=0x7f3c9c0b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0b09a0 0x7f3c9c0b0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 -- 172.21.15.28:0/3632460549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c0b28c0 con 0x7f3c9c14c960 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca1a76640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c14c960 0x7f3c9c14af40 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00ea40 tx=0x7f3c8c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3c92ffd640 1 -- 172.21.15.28:0/3632460549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c009b30 con 0x7f3c9c14c960 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3c92ffd640 1 -- 172.21.15.28:0/3632460549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c009c90 con 0x7f3c9c14c960 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3c92ffd640 1 -- 172.21.15.28:0/3632460549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c00b1c0 con 0x7f3c9c14c960 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c0afec0 con 0x7f3c9c14c960 2023-12-17T06:32:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.158+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c0b1320 con 0x7f3c9c14c960 2023-12-17T06:32:07.170 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.162+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c069440 con 0x7f3c9c14c960 2023-12-17T06:32:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.166+0000 7f3c92ffd640 1 -- 172.21.15.28:0/3632460549 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f3c8c00b320 con 0x7f3c9c14c960 2023-12-17T06:32:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.166+0000 7f3c92ffd640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3c800664a0 0x7f3c80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.166+0000 7f3c92ffd640 1 -- 172.21.15.28:0/3632460549 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c8c0834e0 con 0x7f3c9c14c960 2023-12-17T06:32:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.166+0000 7f3ca2277640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3c800664a0 0x7f3c80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.166+0000 7f3ca2277640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3c800664a0 0x7f3c80068960 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3c98007fd0 tx=0x7f3c98009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.170+0000 7f3c92ffd640 1 -- 172.21.15.28:0/3632460549 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3c8c0508d0 con 0x7f3c9c14c960 2023-12-17T06:32:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.426+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c9c0cb360 con 0x7f3c9c14c960 2023-12-17T06:32:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.430+0000 7f3c92ffd640 1 -- 172.21.15.28:0/3632460549 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f3c8c053ef0 con 0x7f3c9c14c960 2023-12-17T06:32:07.436 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:07.436 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/992095456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":992095456},{"type":"v1","addr":"172.21.15.122:6835","nonce":992095456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3174619236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3174619236},{"type":"v1","addr":"172.21.15.122:6837","nonce":3174619236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4691,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3342817962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3342817962},{"type":"v1","addr":"172.21.15.28:6835","nonce":3342817962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:58:40.841493+0000","modified":"2023-12-16T21:58:42.871930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5193},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5193":{"gid":5193,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/924398795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":924398795},{"type":"v1","addr":"172.21.15.28:6833","nonce":924398795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-17T06:32:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 15 2023-12-17T06:32:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.434+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3c800664a0 msgr2=0x7f3c80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.434+0000 7f3ca44db640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3c800664a0 0x7f3c80068960 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3c98007fd0 tx=0x7f3c98009040 comp rx=0 tx=0).stop 2023-12-17T06:32:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c14c960 msgr2=0x7f3c9c14af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c14c960 0x7f3c9c14af40 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00ea40 tx=0x7f3c8c00ef00 comp rx=0 tx=0).stop 2023-12-17T06:32:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 shutdown_connections 2023-12-17T06:32:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3c800664a0 0x7f3c80068960 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3c9c0b09a0 0x7f3c9c0b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c14c960 0x7f3c9c14af40 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 --2- 172.21.15.28:0/3632460549 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3c9c14c580 0x7f3c9c0b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 >> 172.21.15.28:0/3632460549 conn(0x7f3c9c06a530 msgr2=0x7f3c9c15bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 shutdown_connections 2023-12-17T06:32:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.438+0000 7f3ca44db640 1 -- 172.21.15.28:0/3632460549 wait complete. 2023-12-17T06:32:07.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-17T06:32:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed66c0640 1 Processor -- start 2023-12-17T06:32:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed66c0640 1 -- start start 2023-12-17T06:32:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed66c0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7efed015a450 0x7efed00cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed66c0640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed00ce840 0x7efed0163a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed66c0640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed0072b80 con 0x7efed00715b0 2023-12-17T06:32:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed66c0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed00cecd0 con 0x7efed015a450 2023-12-17T06:32:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed66c0640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed00b2fd0 con 0x7efed00ce840 2023-12-17T06:32:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed4c5d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed00ce840 0x7efed0163a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed4c5d640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed00ce840 0x7efed0163a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:40022/0 (socket says 172.21.15.28:40022) 2023-12-17T06:32:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efed4c5d640 1 -- 172.21.15.28:0/1996151665 learned_addr learned my addr 172.21.15.28:0/1996151665 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efecf7fe640 1 --2- 172.21.15.28:0/1996151665 >> v2:172.21.15.122:3300/0 conn(0x7efed015a450 0x7efed00cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efecf7fe640 1 -- 172.21.15.28:0/1996151665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed00ce840 msgr2=0x7efed0163a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efecf7fe640 1 --2- 172.21.15.28:0/1996151665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed00ce840 0x7efed0163a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efecf7fe640 1 -- 172.21.15.28:0/1996151665 >> v1:172.21.15.28:6789/0 conn(0x7efed00715b0 legacy=0x7efed0071990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-17T06:32:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.606+0000 7efecf7fe640 1 -- 172.21.15.28:0/1996151665 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed00b32b0 con 0x7efed015a450 2023-12-17T06:32:07.616 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efecf7fe640 1 --2- 172.21.15.28:0/1996151665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed015a450 0x7efed00cf860 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7efec400dbe0 tx=0x7efec4020740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26b18ffc0c023dd5 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:07.616 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efeceffd640 1 -- 172.21.15.28:0/1996151665 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec400d2d0 con 0x7efed015a450 2023-12-17T06:32:07.616 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efeceffd640 1 -- 172.21.15.28:0/1996151665 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec40aa4c0 con 0x7efed015a450 2023-12-17T06:32:07.616 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efeceffd640 1 -- 172.21.15.28:0/1996151665 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec40aa850 con 0x7efed015a450 2023-12-17T06:32:07.616 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efed66c0640 1 -- 172.21.15.28:0/1996151665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed015a450 msgr2=0x7efed00cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.617 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efed66c0640 1 --2- 172.21.15.28:0/1996151665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed015a450 0x7efed00cf860 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7efec400dbe0 tx=0x7efec4020740 comp rx=0 tx=0).stop 2023-12-17T06:32:07.617 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efed66c0640 1 -- 172.21.15.28:0/1996151665 shutdown_connections 2023-12-17T06:32:07.617 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efed66c0640 1 --2- 172.21.15.28:0/1996151665 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed00ce840 0x7efed0163a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.617 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efed66c0640 1 --2- 172.21.15.28:0/1996151665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed015a450 0x7efed00cf860 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.617 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efed66c0640 1 -- 172.21.15.28:0/1996151665 >> 172.21.15.28:0/1996151665 conn(0x7efed006d530 msgr2=0x7efed006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:07.618 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efed66c0640 1 -- 172.21.15.28:0/1996151665 shutdown_connections 2023-12-17T06:32:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.610+0000 7efed66c0640 1 -- 172.21.15.28:0/1996151665 wait complete. 2023-12-17T06:32:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed66c0640 1 Processor -- start 2023-12-17T06:32:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed66c0640 1 -- start start 2023-12-17T06:32:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed66c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed00ce840 0x7efed0205f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed66c0640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed015a450 0x7efed013cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed66c0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed0159810 0x7efed0159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed66c0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed013d920 con 0x7efed00ce840 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed66c0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed015a1e0 con 0x7efed0159810 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed66c0640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed0204f00 con 0x7efed015a450 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efecffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed00ce840 0x7efed0205f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efecffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed00ce840 0x7efed0205f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55928/0 (socket says 172.21.15.28:55928) 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efecffff640 1 -- 172.21.15.28:0/1200466036 learned_addr learned my addr 172.21.15.28:0/1200466036 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed4c5d640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed0159810 0x7efed0159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efecf7fe640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed015a450 0x7efed013cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed4c5d640 1 -- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed015a450 msgr2=0x7efed013cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed4c5d640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed015a450 0x7efed013cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed4c5d640 1 -- 172.21.15.28:0/1200466036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed00ce840 msgr2=0x7efed0205f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed4c5d640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed00ce840 0x7efed0205f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed4c5d640 1 -- 172.21.15.28:0/1200466036 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec400d8c0 con 0x7efed0159810 2023-12-17T06:32:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efecffff640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed00ce840 0x7efed0205f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.614+0000 7efed4c5d640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed0159810 0x7efed0159ca0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7efec8009f90 tx=0x7efec80085f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.618+0000 7efeccff9640 1 -- 172.21.15.28:0/1200466036 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8010040 con 0x7efed0159810 2023-12-17T06:32:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.618+0000 7efeccff9640 1 -- 172.21.15.28:0/1200466036 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec8008e90 con 0x7efed0159810 2023-12-17T06:32:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.618+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efed0205150 con 0x7efed0159810 2023-12-17T06:32:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.618+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed0205710 con 0x7efed0159810 2023-12-17T06:32:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.618+0000 7efeccff9640 1 -- 172.21.15.28:0/1200466036 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec8003de0 con 0x7efed0159810 2023-12-17T06:32:07.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.622+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb0005260 con 0x7efed0159810 2023-12-17T06:32:07.628 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.622+0000 7efeccff9640 1 -- 172.21.15.28:0/1200466036 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7efec8004030 con 0x7efed0159810 2023-12-17T06:32:07.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.622+0000 7efeccff9640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efebc0664c0 0x7efebc068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:07.629 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.622+0000 7efeccff9640 1 -- 172.21.15.28:0/1200466036 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efec80840a0 con 0x7efed0159810 2023-12-17T06:32:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.622+0000 7efecffff640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efebc0664c0 0x7efebc068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.622+0000 7efecffff640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efebc0664c0 0x7efebc068980 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7efed02064e0 tx=0x7efec0009110 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:07.635 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.630+0000 7efeccff9640 1 -- 172.21.15.28:0/1200466036 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7efec80516b0 con 0x7efed0159810 2023-12-17T06:32:07.875 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.870+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efeb00050d0 con 0x7efed0159810 2023-12-17T06:32:07.877 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.870+0000 7efeccff9640 1 -- 172.21.15.28:0/1200466036 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16557 (secure 0 0 0) 0x7efec800bce0 con 0x7efed0159810 2023-12-17T06:32:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:07.878 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:42.498586+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:58:38.932657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:58:39.841920+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-17T06:32:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efebc0664c0 msgr2=0x7efebc068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efebc0664c0 0x7efebc068980 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7efed02064e0 tx=0x7efec0009110 comp rx=0 tx=0).stop 2023-12-17T06:32:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed0159810 msgr2=0x7efed0159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed0159810 0x7efed0159ca0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7efec8009f90 tx=0x7efec80085f0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 shutdown_connections 2023-12-17T06:32:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7efebc0664c0 0x7efebc068980 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efed0159810 0x7efed0159ca0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7efed015a450 0x7efed013cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 --2- 172.21.15.28:0/1200466036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed00ce840 0x7efed0205f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 >> 172.21.15.28:0/1200466036 conn(0x7efed006d530 msgr2=0x7efed00afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 shutdown_connections 2023-12-17T06:32:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:07.874+0000 7efed66c0640 1 -- 172.21.15.28:0/1200466036 wait complete. 2023-12-17T06:32:07.907 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-17T06:32:07.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-17T06:32:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f119b31e640 1 Processor -- start 2023-12-17T06:32:08.033 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f119b31e640 1 -- start start 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f119b31e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f119415d490 0x7f11940b6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f119b31e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11940d21f0 con 0x7f11940b51e0 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f119b31e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11940d2740 con 0x7f11940d14c0 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f119b31e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11940d30e0 con 0x7f119415d490 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f11990ba640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f11940b51e0 0x7f11940b55c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33330/0 (socket says 172.21.15.28:33330) 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f11990ba640 1 -- 172.21.15.28:0/2700392897 learned_addr learned my addr 172.21.15.28:0/2700392897 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f11988b9640 1 --2- 172.21.15.28:0/2700392897 >> v2:172.21.15.122:3300/0 conn(0x7f119415d490 0x7f11940b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135897054 0 0) 0x7f11940d21f0 con 0x7f11940b51e0 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1180003530 con 0x7f11940b51e0 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2604098946 0 0) 0x7f1180003530 con 0x7f11940b51e0 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 >> v2:172.21.15.122:3300/0 conn(0x7f119415d490 msgr2=0x7f11940b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:08.036 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f118bfff640 1 --2- 172.21.15.28:0/2700392897 >> v2:172.21.15.122:3300/0 conn(0x7f119415d490 0x7f11940b6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.026+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 >> v1:172.21.15.122:6789/0 conn(0x7f11940d14c0 legacy=0x7f11940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940d2a20 con 0x7f11940b51e0 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f1190001680 con 0x7f11940b51e0 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 >> v1:172.21.15.28:6789/0 conn(0x7f11940b51e0 legacy=0x7f11940b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1180003980 0x7f1180003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f11800047d0 0x7f1180004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f118000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11940d30e0 con 0x7f1180005190 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118000c010 con 0x7f1180003980 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118000c1d0 con 0x7f11800047d0 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11998bb640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f118000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11988b9640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f11800047d0 0x7f1180004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11990ba640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1180003980 0x7f1180003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11998bb640 1 -- 172.21.15.28:0/2700392897 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f11800047d0 msgr2=0x7f1180004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11998bb640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f11800047d0 0x7f1180004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11998bb640 1 -- 172.21.15.28:0/2700392897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1180003980 msgr2=0x7f1180003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11998bb640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1180003980 0x7f1180003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11998bb640 1 -- 172.21.15.28:0/2700392897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1180003530 con 0x7f1180005190 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11998bb640 1 -- 172.21.15.28:0/2700392897 reap_dead start 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f11998bb640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f118000b990 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f118c0044b0 tx=0x7f118c01c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0a8601739f4763 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c0a9680 con 0x7f1180005190 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c0a9800 con 0x7f1180005190 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f118bfff640 1 -- 172.21.15.28:0/2700392897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c0a9bb0 con 0x7f1180005190 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f119b31e640 1 -- 172.21.15.28:0/2700392897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 msgr2=0x7f118000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f119b31e640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f118000b990 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f118c0044b0 tx=0x7f118c01c680 comp rx=0 tx=0).stop 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f119b31e640 1 -- 172.21.15.28:0/2700392897 shutdown_connections 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f119b31e640 1 --2- 172.21.15.28:0/2700392897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f118000b990 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f119b31e640 1 -- 172.21.15.28:0/2700392897 >> 172.21.15.28:0/2700392897 conn(0x7f1194072060 msgr2=0x7f1194072470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f119b31e640 1 -- 172.21.15.28:0/2700392897 shutdown_connections 2023-12-17T06:32:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.030+0000 7f119b31e640 1 -- 172.21.15.28:0/2700392897 wait complete. 2023-12-17T06:32:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 Processor -- start 2023-12-17T06:32:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 -- start start 2023-12-17T06:32:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f1194151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1194152840 0x7f119415d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119415ddb0 0x7f11940cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11940cae80 con 0x7f1180005190 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194152510 con 0x7f119415ddb0 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11940cb120 con 0x7f1194152840 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f1194151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f1194151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55940/0 (socket says 172.21.15.28:55940) 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 -- 172.21.15.28:0/1410433646 learned_addr learned my addr 172.21.15.28:0/1410433646 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11988b9640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1194152840 0x7f119415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 -- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1194152840 msgr2=0x7f119415d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1194152840 0x7f119415d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 -- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119415ddb0 msgr2=0x7f11940cbee0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119415ddb0 0x7f11940cbee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 -- 172.21.15.28:0/1410433646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940cb400 con 0x7f1180005190 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f11990ba640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f1194151fd0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f1190004f00 tx=0x7f1190007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f1189ffb640 1 -- 172.21.15.28:0/1410433646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119000f7b0 con 0x7f1180005190 2023-12-17T06:32:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f1189ffb640 1 -- 172.21.15.28:0/1410433646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119000f950 con 0x7f1180005190 2023-12-17T06:32:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f1189ffb640 1 -- 172.21.15.28:0/1410433646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119000fd00 con 0x7f1180005190 2023-12-17T06:32:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11940cb6e0 con 0x7f1180005190 2023-12-17T06:32:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.034+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11940cc420 con 0x7f1180005190 2023-12-17T06:32:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.038+0000 7f1189ffb640 1 -- 172.21.15.28:0/1410433646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f119001d460 con 0x7f1180005190 2023-12-17T06:32:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.038+0000 7f1189ffb640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f11780664a0 0x7f1178068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.038+0000 7f1189ffb640 1 -- 172.21.15.28:0/1410433646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1190083ef0 con 0x7f1180005190 2023-12-17T06:32:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.038+0000 7f11988b9640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f11780664a0 0x7f1178068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.038+0000 7f11988b9640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f11780664a0 0x7f1178068960 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f11940b8750 tx=0x7f1184008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.042+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1194069440 con 0x7f1180005190 2023-12-17T06:32:08.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.054+0000 7f1189ffb640 1 -- 172.21.15.28:0/1410433646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f11900512d0 con 0x7f1180005190 2023-12-17T06:32:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.326+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11940cbb90 con 0x7f1180005190 2023-12-17T06:32:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.326+0000 7f1189ffb640 1 -- 172.21.15.28:0/1410433646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f11900548f0 con 0x7f1180005190 2023-12-17T06:32:08.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:08.332 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/992095456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":992095456},{"type":"v1","addr":"172.21.15.122:6835","nonce":992095456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3174619236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3174619236},{"type":"v1","addr":"172.21.15.122:6837","nonce":3174619236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4691,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3342817962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3342817962},{"type":"v1","addr":"172.21.15.28:6835","nonce":3342817962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:58:40.841493+0000","modified":"2023-12-16T21:58:42.871930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5193},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5193":{"gid":5193,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/924398795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":924398795},{"type":"v1","addr":"172.21.15.28:6833","nonce":924398795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-17T06:32:08.332 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 15 2023-12-17T06:32:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.330+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f11780664a0 msgr2=0x7f1178068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.330+0000 7f119b31e640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f11780664a0 0x7f1178068960 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f11940b8750 tx=0x7f1184008040 comp rx=0 tx=0).stop 2023-12-17T06:32:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.330+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 msgr2=0x7f1194151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.330+0000 7f119b31e640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f1194151fd0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f1190004f00 tx=0x7f1190007430 comp rx=0 tx=0).stop 2023-12-17T06:32:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.334+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 shutdown_connections 2023-12-17T06:32:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.334+0000 7f119b31e640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f11780664a0 0x7f1178068960 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.334+0000 7f119b31e640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f119415ddb0 0x7f11940cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.334+0000 7f119b31e640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1194152840 0x7f119415d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.334+0000 7f119b31e640 1 --2- 172.21.15.28:0/1410433646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1180005190 0x7f1194151fd0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.334+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 >> 172.21.15.28:0/1410433646 conn(0x7f1194072060 msgr2=0x7f11940b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.334+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 shutdown_connections 2023-12-17T06:32:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.334+0000 7f119b31e640 1 -- 172.21.15.28:0/1410433646 wait complete. 2023-12-17T06:32:08.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-17T06:32:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f862de23640 1 Processor -- start 2023-12-17T06:32:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f862de23640 1 -- start start 2023-12-17T06:32:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f862de23640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f862806eab0 0x7f862813d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f862de23640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f862806ee90 0x7f862813df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f862de23640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862813e470 con 0x7f862815ec80 2023-12-17T06:32:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f862de23640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862815f9f0 con 0x7f862806ee90 2023-12-17T06:32:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f862de23640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862815b900 con 0x7f862806eab0 2023-12-17T06:32:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f8626ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f862806ee90 0x7f862813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f8627fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f862815ec80 0x7f862815f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33346/0 (socket says 172.21.15.28:33346) 2023-12-17T06:32:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f8627fff640 1 -- 172.21.15.28:0/677528028 learned_addr learned my addr 172.21.15.28:0/677528028 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f86277fe640 1 --2- 172.21.15.28:0/677528028 >> v2:172.21.15.122:3300/0 conn(0x7f862806eab0 0x7f862813d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f8626ffd640 1 -- 172.21.15.28:0/677528028 >> v2:172.21.15.122:3300/0 conn(0x7f862806eab0 msgr2=0x7f862813d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f8626ffd640 1 --2- 172.21.15.28:0/677528028 >> v2:172.21.15.122:3300/0 conn(0x7f862806eab0 0x7f862813d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f8626ffd640 1 -- 172.21.15.28:0/677528028 >> v1:172.21.15.28:6789/0 conn(0x7f862815ec80 legacy=0x7f862815f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f8626ffd640 1 -- 172.21.15.28:0/677528028 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862815bbe0 con 0x7f862806ee90 2023-12-17T06:32:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f86267fc640 1 -- 172.21.15.28:0/677528028 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651229011 0 0) 0x7f862813e470 con 0x7f862815ec80 2023-12-17T06:32:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.502+0000 7f86277fe640 1 --2- 172.21.15.28:0/677528028 >> v2:172.21.15.122:3300/0 conn(0x7f862806eab0 0x7f862813d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f8626ffd640 1 --2- 172.21.15.28:0/677528028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806ee90 0x7f862813df30 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f861c0097c0 tx=0x7f861c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879c744f42c0627b server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f86267fc640 1 -- 172.21.15.28:0/677528028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c004230 con 0x7f862806ee90 2023-12-17T06:32:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f86267fc640 1 -- 172.21.15.28:0/677528028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c0a7050 con 0x7f862806ee90 2023-12-17T06:32:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f86267fc640 1 -- 172.21.15.28:0/677528028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c003620 con 0x7f862806ee90 2023-12-17T06:32:08.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f862de23640 1 -- 172.21.15.28:0/677528028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806ee90 msgr2=0x7f862813df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f862de23640 1 --2- 172.21.15.28:0/677528028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806ee90 0x7f862813df30 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f861c0097c0 tx=0x7f861c01c440 comp rx=0 tx=0).stop 2023-12-17T06:32:08.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f862de23640 1 -- 172.21.15.28:0/677528028 shutdown_connections 2023-12-17T06:32:08.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f862de23640 1 --2- 172.21.15.28:0/677528028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806ee90 0x7f862813df30 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f862de23640 1 --2- 172.21.15.28:0/677528028 >> v2:172.21.15.122:3300/0 conn(0x7f862806eab0 0x7f862813d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f862de23640 1 -- 172.21.15.28:0/677528028 >> 172.21.15.28:0/677528028 conn(0x7f862806a930 msgr2=0x7f862806ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:08.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f862de23640 1 -- 172.21.15.28:0/677528028 shutdown_connections 2023-12-17T06:32:08.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.506+0000 7f862de23640 1 -- 172.21.15.28:0/677528028 wait complete. 2023-12-17T06:32:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f862de23640 1 Processor -- start 2023-12-17T06:32:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f862de23640 1 -- start start 2023-12-17T06:32:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f862de23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806eab0 0x7f862814ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f862de23640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f862806ee90 0x7f862813e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f862de23640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862815ec80 0x7f862813eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f862de23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862814f5a0 con 0x7f862806eab0 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f862de23640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862814f840 con 0x7f862815ec80 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f862de23640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862814fb20 con 0x7f862806ee90 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f86277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806eab0 0x7f862814ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f86277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806eab0 0x7f862814ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55962/0 (socket says 172.21.15.28:55962) 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f86277fe640 1 -- 172.21.15.28:0/820112950 learned_addr learned my addr 172.21.15.28:0/820112950 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f8627fff640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862815ec80 0x7f862813eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f86277fe640 1 -- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f862806ee90 msgr2=0x7f862813e620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-17T06:32:08.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f86277fe640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f862806ee90 0x7f862813e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f86277fe640 1 -- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862815ec80 msgr2=0x7f862813eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f86277fe640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862815ec80 0x7f862813eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.510+0000 7f86277fe640 1 -- 172.21.15.28:0/820112950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8618009420 con 0x7f862806eab0 2023-12-17T06:32:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f86277fe640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806eab0 0x7f862814ee00 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f861800ebe0 tx=0x7f861800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f860ffff640 1 -- 172.21.15.28:0/820112950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861800eee0 con 0x7f862806eab0 2023-12-17T06:32:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f860ffff640 1 -- 172.21.15.28:0/820112950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861800cc30 con 0x7f862806eab0 2023-12-17T06:32:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c009420 con 0x7f862806eab0 2023-12-17T06:32:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f860ffff640 1 -- 172.21.15.28:0/820112950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8618013470 con 0x7f862806eab0 2023-12-17T06:32:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86281500e0 con 0x7f862806eab0 2023-12-17T06:32:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8628069440 con 0x7f862806eab0 2023-12-17T06:32:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f860ffff640 1 -- 172.21.15.28:0/820112950 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f86180135d0 con 0x7f862806eab0 2023-12-17T06:32:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f860ffff640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8614066470 0x7f8614068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.518+0000 7f860ffff640 1 -- 172.21.15.28:0/820112950 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8618084060 con 0x7f862806eab0 2023-12-17T06:32:08.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.522+0000 7f8626ffd640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8614066470 0x7f8614068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.526+0000 7f860ffff640 1 -- 172.21.15.28:0/820112950 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f8618051620 con 0x7f862806eab0 2023-12-17T06:32:08.535 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.526+0000 7f8626ffd640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8614066470 0x7f8614068930 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f861c0ad000 tx=0x7f861c0a6000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:08.788 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.782+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86280d1560 con 0x7f862806eab0 2023-12-17T06:32:08.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f860ffff640 1 -- 172.21.15.28:0/820112950 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f8618054c40 con 0x7f862806eab0 2023-12-17T06:32:08.792 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:08.792 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/992095456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":992095456},{"type":"v1","addr":"172.21.15.122:6835","nonce":992095456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3174619236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3174619236},{"type":"v1","addr":"172.21.15.122:6837","nonce":3174619236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4691,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3342817962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3342817962},{"type":"v1","addr":"172.21.15.28:6835","nonce":3342817962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:58:40.841493+0000","modified":"2023-12-16T21:58:42.871930+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5193},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5193":{"gid":5193,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/924398795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":924398795},{"type":"v1","addr":"172.21.15.28:6833","nonce":924398795}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-17T06:32:08.792 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 15 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8614066470 msgr2=0x7f8614068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8614066470 0x7f8614068930 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f861c0ad000 tx=0x7f861c0a6000 comp rx=0 tx=0).stop 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806eab0 msgr2=0x7f862814ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806eab0 0x7f862814ee00 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f861800ebe0 tx=0x7f861800c520 comp rx=0 tx=0).stop 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 shutdown_connections 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8614066470 0x7f8614068930 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f862815ec80 0x7f862813eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f862806ee90 0x7f862813e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 --2- 172.21.15.28:0/820112950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f862806eab0 0x7f862814ee00 secure :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f861800ebe0 tx=0x7f861800c520 comp rx=0 tx=0).stop 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 >> 172.21.15.28:0/820112950 conn(0x7f862806a930 msgr2=0x7f862813b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.786+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 shutdown_connections 2023-12-17T06:32:08.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.790+0000 7f862de23640 1 -- 172.21.15.28:0/820112950 wait complete. 2023-12-17T06:32:08.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-17T06:32:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81efd48640 1 Processor -- start 2023-12-17T06:32:08.964 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81efd48640 1 -- start start 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81efd48640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e815bac0 0x7f81e806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81efd48640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81e80cc8e0 0x7f81e815bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81efd48640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e806fde0 con 0x7f81e806a530 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81efd48640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e80ccdc0 con 0x7f81e80cc8e0 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81efd48640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e806ea60 con 0x7f81e815bac0 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81edae4640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e815bac0 0x7f81e806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81ed2e3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f81e80cc8e0 0x7f81e815bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81ee2e5640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f81e806a530 0x7f81e806d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:45804/0 (socket says 172.21.15.28:45804) 2023-12-17T06:32:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81ee2e5640 1 -- 172.21.15.28:0/1090327628 learned_addr learned my addr 172.21.15.28:0/1090327628 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81edae4640 1 -- 172.21.15.28:0/1090327628 >> v2:172.21.15.122:3300/0 conn(0x7f81e80cc8e0 msgr2=0x7f81e815bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81edae4640 1 --2- 172.21.15.28:0/1090327628 >> v2:172.21.15.122:3300/0 conn(0x7f81e80cc8e0 0x7f81e815bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81edae4640 1 -- 172.21.15.28:0/1090327628 >> v1:172.21.15.122:6789/0 conn(0x7f81e806a530 legacy=0x7f81e806d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81edae4640 1 -- 172.21.15.28:0/1090327628 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e806ed40 con 0x7f81e815bac0 2023-12-17T06:32:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.958+0000 7f81ed2e3640 1 --2- 172.21.15.28:0/1090327628 >> v2:172.21.15.122:3300/0 conn(0x7f81e80cc8e0 0x7f81e815bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.962+0000 7f81edae4640 1 --2- 172.21.15.28:0/1090327628 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e815bac0 0x7f81e806f540 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009d90 tx=0x7f81e806e220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe3bc206919381f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.962+0000 7f81ecae2640 1 -- 172.21.15.28:0/1090327628 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d80a58c0 con 0x7f81e815bac0 2023-12-17T06:32:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.962+0000 7f81ecae2640 1 -- 172.21.15.28:0/1090327628 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d80a5a20 con 0x7f81e815bac0 2023-12-17T06:32:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.962+0000 7f81ecae2640 1 -- 172.21.15.28:0/1090327628 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d80a58c0 con 0x7f81e815bac0 2023-12-17T06:32:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 -- 172.21.15.28:0/1090327628 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e815bac0 msgr2=0x7f81e806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 --2- 172.21.15.28:0/1090327628 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e815bac0 0x7f81e806f540 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f81d8009d90 tx=0x7f81e806e220 comp rx=0 tx=0).stop 2023-12-17T06:32:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 -- 172.21.15.28:0/1090327628 shutdown_connections 2023-12-17T06:32:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 --2- 172.21.15.28:0/1090327628 >> v2:172.21.15.122:3300/0 conn(0x7f81e80cc8e0 0x7f81e815bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 --2- 172.21.15.28:0/1090327628 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e815bac0 0x7f81e806f540 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 -- 172.21.15.28:0/1090327628 >> 172.21.15.28:0/1090327628 conn(0x7f81e80cfe90 msgr2=0x7f81e80d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 -- 172.21.15.28:0/1090327628 shutdown_connections 2023-12-17T06:32:08.974 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 -- 172.21.15.28:0/1090327628 wait complete. 2023-12-17T06:32:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.966+0000 7f81efd48640 1 Processor -- start 2023-12-17T06:32:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 -- start start 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e806a530 0x7f81e8165640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e80cc8e0 0x7f81e8163650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e815bac0 0x7f81e8163b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e8151d00 con 0x7f81e806a530 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e8151a20 con 0x7f81e815bac0 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e81616f0 con 0x7f81e80cc8e0 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e806a530 0x7f81e8165640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e806a530 0x7f81e8165640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55976/0 (socket says 172.21.15.28:55976) 2023-12-17T06:32:08.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 -- 172.21.15.28:0/3111313377 learned_addr learned my addr 172.21.15.28:0/3111313377 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81ed2e3640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e80cc8e0 0x7f81e8163650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81ee2e5640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e815bac0 0x7f81e8163b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 -- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e80cc8e0 msgr2=0x7f81e8163650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e80cc8e0 0x7f81e8163650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 -- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e815bac0 msgr2=0x7f81e8163b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e815bac0 0x7f81e8163b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 -- 172.21.15.28:0/3111313377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc009420 con 0x7f81e806a530 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81edae4640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e806a530 0x7f81e8165640 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f81d801a580 tx=0x7f81d80196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81e67fc640 1 -- 172.21.15.28:0/3111313377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d80a58c0 con 0x7f81e806a530 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81e67fc640 1 -- 172.21.15.28:0/3111313377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d80ad940 con 0x7f81e806a530 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d8009a70 con 0x7f81e806a530 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81e67fc640 1 -- 172.21.15.28:0/3111313377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d80add10 con 0x7f81e806a530 2023-12-17T06:32:08.978 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e8161cb0 con 0x7f81e806a530 2023-12-17T06:32:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e80674d0 con 0x7f81e806a530 2023-12-17T06:32:08.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81e67fc640 1 -- 172.21.15.28:0/3111313377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f81d80b4050 con 0x7f81e806a530 2023-12-17T06:32:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81e67fc640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81d4066470 0x7f81d4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.970+0000 7f81e67fc640 1 -- 172.21.15.28:0/3111313377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81d811eca0 con 0x7f81e806a530 2023-12-17T06:32:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.978+0000 7f81ed2e3640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81d4066470 0x7f81d4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.978+0000 7f81ed2e3640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81d4066470 0x7f81d4068930 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f81e8067ec0 tx=0x7f81dc009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:09.003 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:08.978+0000 7f81e67fc640 1 -- 172.21.15.28:0/3111313377 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f81d80ec380 con 0x7f81e806a530 2023-12-17T06:32:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.230+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81e8161ea0 con 0x7f81e806a530 2023-12-17T06:32:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81e67fc640 1 -- 172.21.15.28:0/3111313377 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f81d80b2090 con 0x7f81e806a530 2023-12-17T06:32:09.247 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:09.247 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-16T21:58:42.498586+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:58:38.932657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:58:39.841920+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81d4066470 msgr2=0x7f81d4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81d4066470 0x7f81d4068930 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f81e8067ec0 tx=0x7f81dc009190 comp rx=0 tx=0).stop 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e806a530 msgr2=0x7f81e8165640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e806a530 0x7f81e8165640 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f81d801a580 tx=0x7f81d80196f0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 shutdown_connections 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f81d4066470 0x7f81d4068930 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81e815bac0 0x7f81e8163b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f81e80cc8e0 0x7f81e8163650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 --2- 172.21.15.28:0/3111313377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81e806a530 0x7f81e8165640 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 >> 172.21.15.28:0/3111313377 conn(0x7f81e80cfe90 msgr2=0x7f81e8154df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 shutdown_connections 2023-12-17T06:32:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.242+0000 7f81efd48640 1 -- 172.21.15.28:0/3111313377 wait complete. 2023-12-17T06:32:09.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-17T06:32:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d2df91640 1 Processor -- start 2023-12-17T06:32:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d2df91640 1 -- start start 2023-12-17T06:32:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d2df91640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d2806ac40 0x7f3d28141270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d2df91640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d28071430 0x7f3d280718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d2df91640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2813e360 con 0x7f3d280cc3d0 2023-12-17T06:32:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d2df91640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d28071e00 con 0x7f3d28071430 2023-12-17T06:32:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d2df91640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d280722f0 con 0x7f3d2806ac40 2023-12-17T06:32:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d26ffd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d2806ac40 0x7f3d28141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d277fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3d280cc3d0 0x7f3d2806a530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:45808/0 (socket says 172.21.15.28:45808) 2023-12-17T06:32:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d277fe640 1 -- 172.21.15.28:0/1269779318 learned_addr learned my addr 172.21.15.28:0/1269779318 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d267fc640 1 -- 172.21.15.28:0/1269779318 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260625362 0 0) 0x7f3d2813e360 con 0x7f3d280cc3d0 2023-12-17T06:32:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d26ffd640 1 -- 172.21.15.28:0/1269779318 >> v2:172.21.15.28:3300/0 conn(0x7f3d28071430 msgr2=0x7f3d280718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d26ffd640 1 --2- 172.21.15.28:0/1269779318 >> v2:172.21.15.28:3300/0 conn(0x7f3d28071430 0x7f3d280718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d26ffd640 1 -- 172.21.15.28:0/1269779318 >> v1:172.21.15.122:6789/0 conn(0x7f3d280cc3d0 legacy=0x7f3d2806a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.398+0000 7f3d26ffd640 1 -- 172.21.15.28:0/1269779318 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d280ccc00 con 0x7f3d2806ac40 2023-12-17T06:32:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d26ffd640 1 --2- 172.21.15.28:0/1269779318 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d2806ac40 0x7f3d28141270 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0097c0 tx=0x7f3d1c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cf5dac74e070c2e server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d267fc640 1 -- 172.21.15.28:0/1269779318 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0ab650 con 0x7f3d2806ac40 2023-12-17T06:32:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d267fc640 1 -- 172.21.15.28:0/1269779318 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0ab7b0 con 0x7f3d2806ac40 2023-12-17T06:32:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d267fc640 1 -- 172.21.15.28:0/1269779318 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0abb40 con 0x7f3d2806ac40 2023-12-17T06:32:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d2df91640 1 -- 172.21.15.28:0/1269779318 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d2806ac40 msgr2=0x7f3d28141270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d2df91640 1 --2- 172.21.15.28:0/1269779318 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d2806ac40 0x7f3d28141270 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0097c0 tx=0x7f3d1c01c450 comp rx=0 tx=0).stop 2023-12-17T06:32:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d2df91640 1 -- 172.21.15.28:0/1269779318 shutdown_connections 2023-12-17T06:32:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d2df91640 1 --2- 172.21.15.28:0/1269779318 >> v2:172.21.15.28:3300/0 conn(0x7f3d28071430 0x7f3d280718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d2df91640 1 --2- 172.21.15.28:0/1269779318 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d2806ac40 0x7f3d28141270 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d2df91640 1 -- 172.21.15.28:0/1269779318 >> 172.21.15.28:0/1269779318 conn(0x7f3d280b1bd0 msgr2=0x7f3d280b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d2df91640 1 -- 172.21.15.28:0/1269779318 shutdown_connections 2023-12-17T06:32:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.402+0000 7f3d2df91640 1 -- 172.21.15.28:0/1269779318 wait complete. 2023-12-17T06:32:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d2df91640 1 Processor -- start 2023-12-17T06:32:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d2df91640 1 -- start start 2023-12-17T06:32:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d2df91640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2806ac40 0x7f3d2813a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d2df91640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d28071430 0x7f3d28164150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d2df91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280cc3d0 0x7f3d28164690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d2df91640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d280b5060 con 0x7f3d280cc3d0 2023-12-17T06:32:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d2df91640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2813a620 con 0x7f3d2806ac40 2023-12-17T06:32:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d2df91640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d280b4000 con 0x7f3d28071430 2023-12-17T06:32:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d28071430 0x7f3d28164150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d277fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2806ac40 0x7f3d2813a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.415 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d27fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280cc3d0 0x7f3d28164690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.415 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d28071430 0x7f3d28164150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:40060/0 (socket says 172.21.15.28:40060) 2023-12-17T06:32:09.415 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 -- 172.21.15.28:0/3016661125 learned_addr learned my addr 172.21.15.28:0/3016661125 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 -- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2806ac40 msgr2=0x7f3d2813a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2806ac40 0x7f3d2813a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 -- 172.21.15.28:0/3016661125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280cc3d0 msgr2=0x7f3d28164690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280cc3d0 0x7f3d28164690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 -- 172.21.15.28:0/3016661125 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c009420 con 0x7f3d28071430 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d277fe640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2806ac40 0x7f3d2813a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.406+0000 7f3d26ffd640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d28071430 0x7f3d28164150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0ad040 tx=0x7f3d1c01ac90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d0ffff640 1 -- 172.21.15.28:0/3016661125 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0b5810 con 0x7f3d28071430 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d0ffff640 1 -- 172.21.15.28:0/3016661125 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0b59b0 con 0x7f3d28071430 2023-12-17T06:32:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d280b42a0 con 0x7f3d28071430 2023-12-17T06:32:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d0ffff640 1 -- 172.21.15.28:0/3016661125 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0b5d60 con 0x7f3d28071430 2023-12-17T06:32:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d280b4860 con 0x7f3d28071430 2023-12-17T06:32:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d280674d0 con 0x7f3d28071430 2023-12-17T06:32:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d0ffff640 1 -- 172.21.15.28:0/3016661125 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f3d1c0ca3e0 con 0x7f3d28071430 2023-12-17T06:32:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d0ffff640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3d14066470 0x7f3d14068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d0ffff640 1 -- 172.21.15.28:0/3016661125 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d1c11f2a0 con 0x7f3d28071430 2023-12-17T06:32:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d277fe640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3d14066470 0x7f3d14068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.410+0000 7f3d277fe640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3d14066470 0x7f3d14068930 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f3d18000f70 tx=0x7f3d18007df0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:09.444 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.438+0000 7f3d0ffff640 1 -- 172.21.15.28:0/3016661125 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f3d1c0ec940 con 0x7f3d28071430 2023-12-17T06:32:09.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.722+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3d2815bb40 con 0x7f3d28071430 2023-12-17T06:32:09.745 INFO:tasks.ceph.mon.a.smithi028.stderr:2023-12-17T06:32:09.738+0000 7fbb08a8a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-17T06:32:09.751 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.746+0000 7f3d0ffff640 1 -- 172.21.15.28:0/3016661125 <== mon.2 v2:172.21.15.122:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3d1c0eff60 con 0x7f3d28071430 2023-12-17T06:32:09.752 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-17T06:32:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3d14066470 msgr2=0x7f3d14068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3d14066470 0x7f3d14068930 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f3d18000f70 tx=0x7f3d18007df0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d28071430 msgr2=0x7f3d28164150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d28071430 0x7f3d28164150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0ad040 tx=0x7f3d1c01ac90 comp rx=0 tx=0).stop 2023-12-17T06:32:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 shutdown_connections 2023-12-17T06:32:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3d14066470 0x7f3d14068930 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d280cc3d0 0x7f3d28164690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f3d28071430 0x7f3d28164150 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 --2- 172.21.15.28:0/3016661125 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d2806ac40 0x7f3d2813a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.750+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 >> 172.21.15.28:0/3016661125 conn(0x7f3d280b1bd0 msgr2=0x7f3d280d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.754+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 shutdown_connections 2023-12-17T06:32:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.754+0000 7f3d2df91640 1 -- 172.21.15.28:0/3016661125 wait complete. 2023-12-17T06:32:09.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-17T06:32:09.885 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2023-12-17T06:32:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f5ad24640 1 Processor -- start 2023-12-17T06:32:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f5ad24640 1 -- start start 2023-12-17T06:32:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f5ad24640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2f54140140 0x7f2f540b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f5ad24640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 0x7f2f5415d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f5ad24640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f54143a50 con 0x7f2f5406a530 2023-12-17T06:32:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f5ad24640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f54142ed0 con 0x7f2f54140140 2023-12-17T06:32:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f5ad24640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f541431b0 con 0x7f2f54140520 2023-12-17T06:32:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f58ac0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2f54140140 0x7f2f540b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f58ac0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2f54140140 0x7f2f540b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56026/0 (socket says 172.21.15.28:56026) 2023-12-17T06:32:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f58ac0640 1 -- 172.21.15.28:0/3670733012 learned_addr learned my addr 172.21.15.28:0/3670733012 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f4bfff640 1 --2- 172.21.15.28:0/3670733012 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 0x7f2f5415d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f58ac0640 1 -- 172.21.15.28:0/3670733012 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 msgr2=0x7f2f5415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f58ac0640 1 --2- 172.21.15.28:0/3670733012 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 0x7f2f5415d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f58ac0640 1 -- 172.21.15.28:0/3670733012 >> v1:172.21.15.122:6789/0 conn(0x7f2f5406a530 legacy=0x7f2f5406a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.890+0000 7f2f58ac0640 1 -- 172.21.15.28:0/3670733012 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f54143490 con 0x7f2f54140140 2023-12-17T06:32:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f58ac0640 1 --2- 172.21.15.28:0/3670733012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f540b4ae0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009870 tx=0x7f2f5001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396b1dcc693700cf server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f4b7fe640 1 -- 172.21.15.28:0/3670733012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500ab600 con 0x7f2f54140140 2023-12-17T06:32:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f4b7fe640 1 -- 172.21.15.28:0/3670733012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f500ab760 con 0x7f2f54140140 2023-12-17T06:32:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f4b7fe640 1 -- 172.21.15.28:0/3670733012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500abaf0 con 0x7f2f54140140 2023-12-17T06:32:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- 172.21.15.28:0/3670733012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 msgr2=0x7f2f540b4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/3670733012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f540b4ae0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009870 tx=0x7f2f5001c440 comp rx=0 tx=0).stop 2023-12-17T06:32:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- 172.21.15.28:0/3670733012 shutdown_connections 2023-12-17T06:32:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/3670733012 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 0x7f2f5415d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/3670733012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f540b4ae0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- 172.21.15.28:0/3670733012 >> 172.21.15.28:0/3670733012 conn(0x7f2f540b1db0 msgr2=0x7f2f540b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- 172.21.15.28:0/3670733012 shutdown_connections 2023-12-17T06:32:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- 172.21.15.28:0/3670733012 wait complete. 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 Processor -- start 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- start start 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f5406a530 0x7f2f541593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f54158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 0x7f2f5413dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5415b130 con 0x7f2f54140140 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5413be50 con 0x7f2f5406a530 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f5ad24640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f541598f0 con 0x7f2f54140520 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f54158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f54158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56036/0 (socket says 172.21.15.28:56036) 2023-12-17T06:32:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f4bfff640 1 -- 172.21.15.28:0/2055976800 learned_addr learned my addr 172.21.15.28:0/2055976800 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f58ac0640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f5406a530 0x7f2f541593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.894+0000 7f2f592c1640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 0x7f2f5413dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f4bfff640 1 -- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 msgr2=0x7f2f5413dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f4bfff640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 0x7f2f5413dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f4bfff640 1 -- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f5406a530 msgr2=0x7f2f541593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f4bfff640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f5406a530 0x7f2f541593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f4bfff640 1 -- 172.21.15.28:0/2055976800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f500094d0 con 0x7f2f54140140 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f4bfff640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f54158ae0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400d5f0 tx=0x7f2f4400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f497fa640 1 -- 172.21.15.28:0/2055976800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f440125f0 con 0x7f2f54140140 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f497fa640 1 -- 172.21.15.28:0/2055976800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f44012750 con 0x7f2f54140140 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f54159b90 con 0x7f2f54140140 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f497fa640 1 -- 172.21.15.28:0/2055976800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44012ae0 con 0x7f2f54140140 2023-12-17T06:32:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5415a150 con 0x7f2f54140140 2023-12-17T06:32:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f540674d0 con 0x7f2f54140140 2023-12-17T06:32:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f497fa640 1 -- 172.21.15.28:0/2055976800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f2f44012c40 con 0x7f2f54140140 2023-12-17T06:32:09.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f497fa640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f40066470 0x7f2f40068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:09.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f497fa640 1 -- 172.21.15.28:0/2055976800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2f440843d0 con 0x7f2f54140140 2023-12-17T06:32:09.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f58ac0640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f40066470 0x7f2f40068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:09.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.898+0000 7f2f58ac0640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f40066470 0x7f2f40068930 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2f500af000 tx=0x7f2f500a7000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:09.909 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:09.902+0000 7f2f497fa640 1 -- 172.21.15.28:0/2055976800 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f2f440517f0 con 0x7f2f54140140 2023-12-17T06:32:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.086+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2f5406eb20 con 0x7f2f54140140 2023-12-17T06:32:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.742+0000 7f2f497fa640 1 -- 172.21.15.28:0/2055976800 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2f44054e10 con 0x7f2f54140140 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f40066470 msgr2=0x7f2f40068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f40066470 0x7f2f40068930 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2f500af000 tx=0x7f2f500a7000 comp rx=0 tx=0).stop 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 msgr2=0x7f2f54158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f54158ae0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400d5f0 tx=0x7f2f4400dab0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 shutdown_connections 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f2f40066470 0x7f2f40068930 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f2f54140520 0x7f2f5413dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f54140140 0x7f2f54158ae0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 --2- 172.21.15.28:0/2055976800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f5406a530 0x7f2f541593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 >> 172.21.15.28:0/2055976800 conn(0x7f2f540b1db0 msgr2=0x7f2f540b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 shutdown_connections 2023-12-17T06:32:10.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.746+0000 7f2f5ad24640 1 -- 172.21.15.28:0/2055976800 wait complete. 2023-12-17T06:32:10.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-17T06:32:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4e8ed640 1 Processor -- start 2023-12-17T06:32:10.913 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4e8ed640 1 -- start start 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4e8ed640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cd570 0x7fdf480c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4e8ed640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdf480cb0c0 0x7fdf48070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4e8ed640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf48159f80 con 0x7fdf48159ba0 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4e8ed640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf480cb550 con 0x7fdf480cb0c0 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4e8ed640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf480ce460 con 0x7fdf480cd570 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4ce8a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdf480cb0c0 0x7fdf48070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf47fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdf48159ba0 0x7fdf480cce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33348/0 (socket says 172.21.15.28:33348) 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf47fff640 1 -- 172.21.15.28:0/2985022067 learned_addr learned my addr 172.21.15.28:0/2985022067 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf477fe640 1 --2- 172.21.15.28:0/2985022067 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cd570 0x7fdf480c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4ce8a640 1 -- 172.21.15.28:0/2985022067 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cd570 msgr2=0x7fdf480c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4ce8a640 1 --2- 172.21.15.28:0/2985022067 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cd570 0x7fdf480c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.906+0000 7fdf4ce8a640 1 -- 172.21.15.28:0/2985022067 >> v1:172.21.15.28:6789/0 conn(0x7fdf48159ba0 legacy=0x7fdf480cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf46ffd640 1 -- 172.21.15.28:0/2985022067 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27376901 0 0) 0x7fdf48159f80 con 0x7fdf48159ba0 2023-12-17T06:32:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4ce8a640 1 -- 172.21.15.28:0/2985022067 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf480cda30 con 0x7fdf480cb0c0 2023-12-17T06:32:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4ce8a640 1 --2- 172.21.15.28:0/2985022067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cb0c0 0x7fdf48070580 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009d90 tx=0x7fdf3801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7259d2ad04c76c4 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf46ffd640 1 -- 172.21.15.28:0/2985022067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf380a5960 con 0x7fdf480cb0c0 2023-12-17T06:32:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf46ffd640 1 -- 172.21.15.28:0/2985022067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf380a5ac0 con 0x7fdf480cb0c0 2023-12-17T06:32:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf46ffd640 1 -- 172.21.15.28:0/2985022067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf380a5e50 con 0x7fdf480cb0c0 2023-12-17T06:32:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/2985022067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cb0c0 msgr2=0x7fdf48070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/2985022067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cb0c0 0x7fdf48070580 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009d90 tx=0x7fdf3801c920 comp rx=0 tx=0).stop 2023-12-17T06:32:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/2985022067 shutdown_connections 2023-12-17T06:32:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/2985022067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cb0c0 0x7fdf48070580 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/2985022067 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cd570 0x7fdf480c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/2985022067 >> 172.21.15.28:0/2985022067 conn(0x7fdf480b0c50 msgr2=0x7fdf480b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/2985022067 shutdown_connections 2023-12-17T06:32:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.910+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/2985022067 wait complete. 2023-12-17T06:32:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 Processor -- start 2023-12-17T06:32:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 -- start start 2023-12-17T06:32:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cb0c0 0x7fdf4809b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cd570 0x7fdf480ada40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf48159ba0 0x7fdf4809cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4809b800 con 0x7fdf480cd570 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf480adf80 con 0x7fdf48159ba0 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4809ba10 con 0x7fdf480cb0c0 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cd570 0x7fdf480ada40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cd570 0x7fdf480ada40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56050/0 (socket says 172.21.15.28:56050) 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 -- 172.21.15.28:0/3591446881 learned_addr learned my addr 172.21.15.28:0/3591446881 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf47fff640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cb0c0 0x7fdf4809b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4ce8a640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf48159ba0 0x7fdf4809cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 -- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cb0c0 msgr2=0x7fdf4809b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cb0c0 0x7fdf4809b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 -- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf48159ba0 msgr2=0x7fdf4809cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf48159ba0 0x7fdf4809cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 -- 172.21.15.28:0/3591446881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf38009a70 con 0x7fdf480cd570 2023-12-17T06:32:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf47fff640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cb0c0 0x7fdf4809b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-17T06:32:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf477fe640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cd570 0x7fdf480ada40 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000d800 tx=0x7fdf3000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf44ff9640 1 -- 172.21.15.28:0/3591446881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3000b6c0 con 0x7fdf480cd570 2023-12-17T06:32:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf44ff9640 1 -- 172.21.15.28:0/3591446881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3000b820 con 0x7fdf480cd570 2023-12-17T06:32:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf44ff9640 1 -- 172.21.15.28:0/3591446881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3000bb90 con 0x7fdf480cd570 2023-12-17T06:32:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4809bcf0 con 0x7fdf480cd570 2023-12-17T06:32:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4809c2b0 con 0x7fdf480cd570 2023-12-17T06:32:10.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4804e850 con 0x7fdf480cd570 2023-12-17T06:32:10.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.914+0000 7fdf44ff9640 1 -- 172.21.15.28:0/3591446881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fdf3000bcf0 con 0x7fdf480cd570 2023-12-17T06:32:10.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.918+0000 7fdf44ff9640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdf28066470 0x7fdf28068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:10.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.918+0000 7fdf44ff9640 1 -- 172.21.15.28:0/3591446881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdf30084470 con 0x7fdf480cd570 2023-12-17T06:32:10.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.918+0000 7fdf47fff640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdf28066470 0x7fdf28068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:10.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.918+0000 7fdf47fff640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdf28066470 0x7fdf28068930 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdf40007fd0 tx=0x7fdf40009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:10.922+0000 7fdf44ff9640 1 -- 172.21.15.28:0/3591446881 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fdf30051890 con 0x7fdf480cd570 2023-12-17T06:32:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.070+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdf480693d0 con 0x7fdf480cd570 2023-12-17T06:32:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.754+0000 7fdf44ff9640 1 -- 172.21.15.28:0/3591446881 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v38) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdf30011340 con 0x7fdf480cd570 2023-12-17T06:32:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdf28066470 msgr2=0x7fdf28068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdf28066470 0x7fdf28068930 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdf40007fd0 tx=0x7fdf40009040 comp rx=0 tx=0).stop 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cd570 msgr2=0x7fdf480ada40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cd570 0x7fdf480ada40 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000d800 tx=0x7fdf3000dcc0 comp rx=0 tx=0).stop 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 shutdown_connections 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fdf28066470 0x7fdf28068930 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf48159ba0 0x7fdf4809cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf480cd570 0x7fdf480ada40 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 --2- 172.21.15.28:0/3591446881 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fdf480cb0c0 0x7fdf4809b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 >> 172.21.15.28:0/3591446881 conn(0x7fdf480b0c50 msgr2=0x7fdf481575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 shutdown_connections 2023-12-17T06:32:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:11.758+0000 7fdf4e8ed640 1 -- 172.21.15.28:0/3591446881 wait complete. 2023-12-17T06:32:11.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-17T06:32:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f493546b640 1 Processor -- start 2023-12-17T06:32:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f493546b640 1 -- start start 2023-12-17T06:32:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f493546b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4930161400 0x7f493006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f493546b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f493006cab0 0x7f49300afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f493546b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49300cc690 con 0x7f49300b03b0 2023-12-17T06:32:12.017 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f493546b640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49300b0820 con 0x7f4930161400 2023-12-17T06:32:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f493546b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49301617e0 con 0x7f493006cab0 2023-12-17T06:32:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f492f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f49300b03b0 0x7f49300b1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:45846/0 (socket says 172.21.15.28:45846) 2023-12-17T06:32:12.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.010+0000 7f492f7fe640 1 -- 172.21.15.28:0/1604166118 learned_addr learned my addr 172.21.15.28:0/1604166118 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345363918 0 0) 0x7f49300cc690 con 0x7f49300b03b0 2023-12-17T06:32:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4918003530 con 0x7f49300b03b0 2023-12-17T06:32:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 <== mon.1 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 206373964 0 0) 0x7f4918003530 con 0x7f49300b03b0 2023-12-17T06:32:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f493006cab0 msgr2=0x7f49300afe70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-17T06:32:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f493006cab0 0x7f49300afe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 >> v2:172.21.15.28:3300/0 conn(0x7f4930161400 msgr2=0x7f493006c570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-17T06:32:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 --2- 172.21.15.28:0/1604166118 >> v2:172.21.15.28:3300/0 conn(0x7f4930161400 0x7f493006c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493006cfc0 con 0x7f49300b03b0 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 <== mon.1 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f4928001a00 con 0x7f49300b03b0 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 >> v1:172.21.15.122:6789/0 conn(0x7f49300b03b0 legacy=0x7f49300b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.014+0000 7f492dffb640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 0x7f4918003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492dffb640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49180047d0 0x7f4918004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492dffb640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4918005190 0x7f491800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300b0820 con 0x7f49180047d0 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49301617e0 con 0x7f4918005190 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491800c110 con 0x7f49180039b0 2023-12-17T06:32:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492f7fe640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4918005190 0x7f491800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492effd640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 0x7f4918003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492effd640 1 -- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4918005190 msgr2=0x7f491800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492effd640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4918005190 0x7f491800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492effd640 1 -- 172.21.15.28:0/1604166118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49180047d0 msgr2=0x7f4918004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492e7fc640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49180047d0 0x7f4918004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492effd640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49180047d0 0x7f4918004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492effd640 1 -- 172.21.15.28:0/1604166118 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4918003530 con 0x7f49180039b0 2023-12-17T06:32:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.018+0000 7f492e7fc640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49180047d0 0x7f4918004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-17T06:32:12.037 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.030+0000 7f492effd640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 0x7f4918003d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f49200097e0 tx=0x7f492001c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98817f7159a85349 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:12.040 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49200ab600 con 0x7f49180039b0 2023-12-17T06:32:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49200a7050 con 0x7f49180039b0 2023-12-17T06:32:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f492dffb640 1 -- 172.21.15.28:0/1604166118 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49200ab9c0 con 0x7f49180039b0 2023-12-17T06:32:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f493546b640 1 -- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 msgr2=0x7f4918003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f493546b640 1 --2- 172.21.15.28:0/1604166118 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 0x7f4918003d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f49200097e0 tx=0x7f492001c300 comp rx=0 tx=0).stop 2023-12-17T06:32:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f492f7fe640 1 -- 172.21.15.28:0/1604166118 reap_dead start 2023-12-17T06:32:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f493546b640 1 -- 172.21.15.28:0/1604166118 shutdown_connections 2023-12-17T06:32:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f493546b640 1 -- 172.21.15.28:0/1604166118 >> 172.21.15.28:0/1604166118 conn(0x7f49300aea50 msgr2=0x7f49300722c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f493546b640 1 -- 172.21.15.28:0/1604166118 shutdown_connections 2023-12-17T06:32:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.034+0000 7f493546b640 1 -- 172.21.15.28:0/1604166118 wait complete. 2023-12-17T06:32:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f493546b640 1 Processor -- start 2023-12-17T06:32:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f493546b640 1 -- start start 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f493546b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 0x7f493013d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f493546b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930145270 0x7f4930145b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f493546b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4930146070 0x7f493014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f493546b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4930146800 con 0x7f4930145270 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f493546b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4930144de0 con 0x7f4930146070 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f493546b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4930146520 con 0x7f49180039b0 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930145270 0x7f4930145b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930145270 0x7f4930145b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56068/0 (socket says 172.21.15.28:56068) 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 -- 172.21.15.28:0/3485100758 learned_addr learned my addr 172.21.15.28:0/3485100758 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492f7fe640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4930146070 0x7f493014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 -- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 msgr2=0x7f493013d3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-17T06:32:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 0x7f493013d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 -- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4930146070 msgr2=0x7f493014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4930146070 0x7f493014a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 -- 172.21.15.28:0/3485100758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4930144790 con 0x7f4930145270 2023-12-17T06:32:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.038+0000 7f492e7fc640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930145270 0x7f4930145b30 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f492400c5a0 tx=0x7f492400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.042+0000 7f49177fe640 1 -- 172.21.15.28:0/3485100758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924011800 con 0x7f4930145270 2023-12-17T06:32:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.042+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4930144a70 con 0x7f4930145270 2023-12-17T06:32:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.042+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49301456f0 con 0x7f4930145270 2023-12-17T06:32:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.042+0000 7f49177fe640 1 -- 172.21.15.28:0/3485100758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4924011960 con 0x7f4930145270 2023-12-17T06:32:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.042+0000 7f49177fe640 1 -- 172.21.15.28:0/3485100758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924011d10 con 0x7f4930145270 2023-12-17T06:32:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.042+0000 7f49177fe640 1 -- 172.21.15.28:0/3485100758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f4924018360 con 0x7f4930145270 2023-12-17T06:32:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.046+0000 7f49177fe640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f491c066470 0x7f491c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.046+0000 7f49177fe640 1 -- 172.21.15.28:0/3485100758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f4924083d00 con 0x7f4930145270 2023-12-17T06:32:12.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.058+0000 7f492effd640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f491c066470 0x7f491c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:12.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.062+0000 7f492effd640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f491c066470 0x7f491c068930 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f49200ad000 tx=0x7f4920004780 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:12.102 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.094+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4910005260 con 0x7f4930145270 2023-12-17T06:32:12.112 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.106+0000 7f49177fe640 1 -- 172.21.15.28:0/3485100758 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f49240512d0 con 0x7f4930145270 2023-12-17T06:32:12.683 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.670+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4910005510 con 0x7f4930145270 2023-12-17T06:32:12.763 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.758+0000 7f49177fe640 1 -- 172.21.15.28:0/3485100758 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v39) v1 ==== 169+0+0 (secure 0 0 0) 0x7f492401f020 con 0x7f4930145270 2023-12-17T06:32:12.763 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2023-12-17T06:32:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f491c066470 msgr2=0x7f491c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f491c066470 0x7f491c068930 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f49200ad000 tx=0x7f4920004780 comp rx=0 tx=0).stop 2023-12-17T06:32:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930145270 msgr2=0x7f4930145b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930145270 0x7f4930145b30 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f492400c5a0 tx=0x7f492400ca60 comp rx=0 tx=0).stop 2023-12-17T06:32:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 shutdown_connections 2023-12-17T06:32:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f491c066470 0x7f491c068930 secure :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f49200ad000 tx=0x7f4920004780 comp rx=0 tx=0).stop 2023-12-17T06:32:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4930146070 0x7f493014a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4930145270 0x7f4930145b30 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 --2- 172.21.15.28:0/3485100758 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f49180039b0 0x7f493013d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.790+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 >> 172.21.15.28:0/3485100758 conn(0x7f49300aea50 msgr2=0x7f4930070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:12.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.794+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 shutdown_connections 2023-12-17T06:32:12.799 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:12.794+0000 7f493546b640 1 -- 172.21.15.28:0/3485100758 wait complete. 2023-12-17T06:32:12.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-17T06:32:13.115 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cfcc51640 1 Processor -- start 2023-12-17T06:32:13.116 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cfcc51640 1 -- start start 2023-12-17T06:32:13.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cfcc51640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7cf8159290 0x7f7cf80721a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cfcc51640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf80ac160 con 0x7f7cf80abd00 2023-12-17T06:32:13.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cfcc51640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf8071970 con 0x7f7cf80b1f30 2023-12-17T06:32:13.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cfcc51640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf806c760 con 0x7f7cf8159290 2023-12-17T06:32:13.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cf6ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7cf80abd00 0x7f7cf8070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33356/0 (socket says 172.21.15.28:33356) 2023-12-17T06:32:13.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cf6ffd640 1 -- 172.21.15.28:0/277038185 learned_addr learned my addr 172.21.15.28:0/277038185 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:13.117 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cf77fe640 1 --2- 172.21.15.28:0/277038185 >> v2:172.21.15.122:3300/0 conn(0x7f7cf8159290 0x7f7cf80721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:13.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cf77fe640 1 -- 172.21.15.28:0/277038185 >> v1:172.21.15.122:6789/0 conn(0x7f7cf80b1f30 legacy=0x7f7cf806cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:13.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cf77fe640 1 -- 172.21.15.28:0/277038185 >> v1:172.21.15.28:6789/0 conn(0x7f7cf80abd00 legacy=0x7f7cf8070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:13.118 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.110+0000 7f7cf77fe640 1 -- 172.21.15.28:0/277038185 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf8071690 con 0x7f7cf8159290 2023-12-17T06:32:13.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.118+0000 7f7cf77fe640 1 --2- 172.21.15.28:0/277038185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cf8159290 0x7f7cf80721a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80097c0 tx=0x7f7ce801e630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d61862525d7c04 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:13.127 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.122+0000 7f7cf67fc640 1 -- 172.21.15.28:0/277038185 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80aa640 con 0x7f7cf8159290 2023-12-17T06:32:13.127 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.122+0000 7f7cf67fc640 1 -- 172.21.15.28:0/277038185 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce80aa7a0 con 0x7f7cf8159290 2023-12-17T06:32:13.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.122+0000 7f7cfcc51640 1 -- 172.21.15.28:0/277038185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cf8159290 msgr2=0x7f7cf80721a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:13.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.122+0000 7f7cfcc51640 1 --2- 172.21.15.28:0/277038185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cf8159290 0x7f7cf80721a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80097c0 tx=0x7f7ce801e630 comp rx=0 tx=0).stop 2023-12-17T06:32:13.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.122+0000 7f7cfcc51640 1 -- 172.21.15.28:0/277038185 shutdown_connections 2023-12-17T06:32:13.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.122+0000 7f7cfcc51640 1 --2- 172.21.15.28:0/277038185 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cf8159290 0x7f7cf80721a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.128 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.122+0000 7f7cfcc51640 1 -- 172.21.15.28:0/277038185 >> 172.21.15.28:0/277038185 conn(0x7f7cf80b0920 msgr2=0x7f7cf80b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:13.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 -- 172.21.15.28:0/277038185 shutdown_connections 2023-12-17T06:32:13.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 -- 172.21.15.28:0/277038185 wait complete. 2023-12-17T06:32:13.137 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 Processor -- start 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 -- start start 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80abd00 0x7f7cf8162a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7cf80b1f30 0x7f7cf8142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cf8159290 0x7f7cf81fe280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8162fc0 con 0x7f7cf80abd00 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf81fedf0 con 0x7f7cf8159290 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cfcc51640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf81ff0d0 con 0x7f7cf80b1f30 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cf77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80abd00 0x7f7cf8162a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cf77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80abd00 0x7f7cf8162a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56078/0 (socket says 172.21.15.28:56078) 2023-12-17T06:32:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cf77fe640 1 -- 172.21.15.28:0/2656017065 learned_addr learned my addr 172.21.15.28:0/2656017065 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.130+0000 7f7cf6ffd640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7cf80b1f30 0x7f7cf8142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cf77fe640 1 -- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7cf80b1f30 msgr2=0x7f7cf8142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cf77fe640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7cf80b1f30 0x7f7cf8142590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cf77fe640 1 -- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cf8159290 msgr2=0x7f7cf81fe280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-17T06:32:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cf77fe640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cf8159290 0x7f7cf81fe280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cf77fe640 1 -- 172.21.15.28:0/2656017065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce8009420 con 0x7f7cf80abd00 2023-12-17T06:32:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cf77fe640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80abd00 0x7f7cf8162a80 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80097c0 tx=0x7f7ce801b3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cdffff640 1 -- 172.21.15.28:0/2656017065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b28c0 con 0x7f7cf80abd00 2023-12-17T06:32:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cdffff640 1 -- 172.21.15.28:0/2656017065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce80b2a20 con 0x7f7cf80abd00 2023-12-17T06:32:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cdffff640 1 -- 172.21.15.28:0/2656017065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b2d90 con 0x7f7cf80abd00 2023-12-17T06:32:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cfcc51640 1 -- 172.21.15.28:0/2656017065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf8162280 con 0x7f7cf80abd00 2023-12-17T06:32:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.134+0000 7f7cfcc51640 1 -- 172.21.15.28:0/2656017065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf814db70 con 0x7f7cf80abd00 2023-12-17T06:32:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.138+0000 7f7cdffff640 1 -- 172.21.15.28:0/2656017065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7ce80b8070 con 0x7f7cf80abd00 2023-12-17T06:32:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.138+0000 7f7cdffff640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ce4066540 0x7f7ce4068a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.138+0000 7f7cdffff640 1 -- 172.21.15.28:0/2656017065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7ce811e1e0 con 0x7f7cf80abd00 2023-12-17T06:32:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.138+0000 7f7cf6ffd640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ce4066540 0x7f7ce4068a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.138+0000 7f7cf6ffd640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ce4066540 0x7f7ce4068a00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80b02c0 tx=0x7f7cf0006200 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:13.151 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.146+0000 7f7cfcc51640 1 -- 172.21.15.28:0/2656017065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf809d080 con 0x7f7cf80abd00 2023-12-17T06:32:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.158+0000 7f7cdffff640 1 -- 172.21.15.28:0/2656017065 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7ce80eb9f0 con 0x7f7cf80abd00 2023-12-17T06:32:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.486+0000 7f7cfcc51640 1 -- 172.21.15.28:0/2656017065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7cf80ce510 con 0x7f7cf80abd00 2023-12-17T06:32:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.494+0000 7f7cdffff640 1 -- 172.21.15.28:0/2656017065 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7ce80bf070 con 0x7f7cf80abd00 2023-12-17T06:32:13.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 -- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ce4066540 msgr2=0x7f7ce4068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:13.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ce4066540 0x7f7ce4068a00 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80b02c0 tx=0x7f7cf0006200 comp rx=0 tx=0).stop 2023-12-17T06:32:13.508 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 -- 172.21.15.28:0/2656017065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80abd00 msgr2=0x7f7cf8162a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80abd00 0x7f7cf8162a80 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80097c0 tx=0x7f7ce801b3d0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 -- 172.21.15.28:0/2656017065 shutdown_connections 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7ce4066540 0x7f7ce4068a00 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7cf8159290 0x7f7cf81fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7cf80b1f30 0x7f7cf8142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 --2- 172.21.15.28:0/2656017065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80abd00 0x7f7cf8162a80 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 -- 172.21.15.28:0/2656017065 >> 172.21.15.28:0/2656017065 conn(0x7f7cf80b0920 msgr2=0x7f7cf806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 -- 172.21.15.28:0/2656017065 shutdown_connections 2023-12-17T06:32:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.502+0000 7f7cddffb640 1 -- 172.21.15.28:0/2656017065 wait complete. 2023-12-17T06:32:13.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic' 2023-12-17T06:32:13.853 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.846+0000 7f38b425c640 1 Processor -- start 2023-12-17T06:32:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.846+0000 7f38b425c640 1 -- start start 2023-12-17T06:32:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b425c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f38ac0704f0 0x7f38ac071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b425c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0ccc80 0x7f38ac0cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b425c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ac14b510 con 0x7f38ac0b32e0 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b425c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ac071cf0 con 0x7f38ac0704f0 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b425c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac071fd0 con 0x7f38ac0ccc80 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b27f9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0ccc80 0x7f38ac0cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b27f9640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0ccc80 0x7f38ac0cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:40164/0 (socket says 172.21.15.28:40164) 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b27f9640 1 -- 172.21.15.28:0/3888283583 learned_addr learned my addr 172.21.15.28:0/3888283583 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b27f9640 1 -- 172.21.15.28:0/3888283583 >> v2:172.21.15.122:3300/0 conn(0x7f38ac0704f0 msgr2=0x7f38ac071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b27f9640 1 --2- 172.21.15.28:0/3888283583 >> v2:172.21.15.122:3300/0 conn(0x7f38ac0704f0 0x7f38ac071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b27f9640 1 -- 172.21.15.28:0/3888283583 >> v1:172.21.15.122:6789/0 conn(0x7f38ac0b32e0 legacy=0x7f38ac0b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.850+0000 7f38b27f9640 1 -- 172.21.15.28:0/3888283583 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac0722b0 con 0x7f38ac0ccc80 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b27f9640 1 --2- 172.21.15.28:0/3888283583 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0ccc80 0x7f38ac0cd110 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f389c009870 tx=0x7f389c01c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f143184092d1cdc1 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b0ff6640 1 -- 172.21.15.28:0/3888283583 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c0ab640 con 0x7f38ac0ccc80 2023-12-17T06:32:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b0ff6640 1 -- 172.21.15.28:0/3888283583 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389c0a7070 con 0x7f38ac0ccc80 2023-12-17T06:32:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b425c640 1 -- 172.21.15.28:0/3888283583 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0ccc80 msgr2=0x7f38ac0cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b425c640 1 --2- 172.21.15.28:0/3888283583 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0ccc80 0x7f38ac0cd110 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f389c009870 tx=0x7f389c01c460 comp rx=0 tx=0).stop 2023-12-17T06:32:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b425c640 1 -- 172.21.15.28:0/3888283583 shutdown_connections 2023-12-17T06:32:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b425c640 1 --2- 172.21.15.28:0/3888283583 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0ccc80 0x7f38ac0cd110 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b425c640 1 --2- 172.21.15.28:0/3888283583 >> v2:172.21.15.122:3300/0 conn(0x7f38ac0704f0 0x7f38ac071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b425c640 1 -- 172.21.15.28:0/3888283583 >> 172.21.15.28:0/3888283583 conn(0x7f38ac06c2c0 msgr2=0x7f38ac06c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:13.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b425c640 1 -- 172.21.15.28:0/3888283583 shutdown_connections 2023-12-17T06:32:13.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.854+0000 7f38b425c640 1 -- 172.21.15.28:0/3888283583 wait complete. 2023-12-17T06:32:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b425c640 1 Processor -- start 2023-12-17T06:32:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b425c640 1 -- start start 2023-12-17T06:32:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b425c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ac0704f0 0x7f38ac161b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b425c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0b32e0 0x7f38ac13a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b425c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac164750 0x7f38ac15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b425c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac15f470 con 0x7f38ac164750 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b425c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac162460 con 0x7f38ac0704f0 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b425c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac162740 con 0x7f38ac0b32e0 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac164750 0x7f38ac15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac164750 0x7f38ac15b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56080/0 (socket says 172.21.15.28:56080) 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 -- 172.21.15.28:0/2277686288 learned_addr learned my addr 172.21.15.28:0/2277686288 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 -- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0b32e0 msgr2=0x7f38ac13a670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-17T06:32:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0b32e0 0x7f38ac13a670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 -- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ac0704f0 msgr2=0x7f38ac161b30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-17T06:32:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ac0704f0 0x7f38ac161b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 -- 172.21.15.28:0/2277686288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c0094d0 con 0x7f38ac164750 2023-12-17T06:32:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.858+0000 7f38b27f9640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac164750 0x7f38ac15b2f0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f389c0ad040 tx=0x7f389c01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.862+0000 7f38a27fc640 1 -- 172.21.15.28:0/2277686288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c0b46b0 con 0x7f38ac164750 2023-12-17T06:32:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.862+0000 7f38a27fc640 1 -- 172.21.15.28:0/2277686288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389c0b4850 con 0x7f38ac164750 2023-12-17T06:32:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.862+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ac162a20 con 0x7f38ac164750 2023-12-17T06:32:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.862+0000 7f38a27fc640 1 -- 172.21.15.28:0/2277686288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c0b4c00 con 0x7f38ac164750 2023-12-17T06:32:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.862+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ac162070 con 0x7f38ac164750 2023-12-17T06:32:13.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.862+0000 7f38a27fc640 1 -- 172.21.15.28:0/2277686288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f389c0aa420 con 0x7f38ac164750 2023-12-17T06:32:13.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.862+0000 7f38a27fc640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38980666a0 0x7f3898068b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:13.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.862+0000 7f38a27fc640 1 -- 172.21.15.28:0/2277686288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f389c11e210 con 0x7f38ac164750 2023-12-17T06:32:13.871 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.866+0000 7f38b1ff8640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38980666a0 0x7f3898068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:13.884 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.866+0000 7f38b1ff8640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38980666a0 0x7f3898068b60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f38a400abb0 tx=0x7f38a4009250 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:13.884 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.866+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3890005260 con 0x7f38ac164750 2023-12-17T06:32:13.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:13.890+0000 7f38a27fc640 1 -- 172.21.15.28:0/2277686288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f389c0ebb20 con 0x7f38ac164750 2023-12-17T06:32:15.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.238+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic"]} v 0) v1 -- 0x7f3890005510 con 0x7f38ac164750 2023-12-17T06:32:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.442+0000 7f38a27fc640 1 -- 172.21.15.28:0/2277686288 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev.test_generic"]}]=0 v15519) v1 ==== 144+0+0 (secure 0 0 0) 0x7f389c0ef140 con 0x7f38ac164750 2023-12-17T06:32:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.486+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38980666a0 msgr2=0x7f3898068b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.486+0000 7f38b425c640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38980666a0 0x7f3898068b60 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f38a400abb0 tx=0x7f38a4009250 comp rx=0 tx=0).stop 2023-12-17T06:32:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.486+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac164750 msgr2=0x7f38ac15b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.486+0000 7f38b425c640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac164750 0x7f38ac15b2f0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f389c0ad040 tx=0x7f389c01aca0 comp rx=0 tx=0).stop 2023-12-17T06:32:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.486+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 shutdown_connections 2023-12-17T06:32:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.486+0000 7f38b425c640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f38980666a0 0x7f3898068b60 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.490+0000 7f38b425c640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac164750 0x7f38ac15b2f0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.490+0000 7f38b425c640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f38ac0b32e0 0x7f38ac13a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.490+0000 7f38b425c640 1 --2- 172.21.15.28:0/2277686288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38ac0704f0 0x7f38ac161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.490+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 >> 172.21.15.28:0/2277686288 conn(0x7f38ac06c2c0 msgr2=0x7f38ac149230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.490+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 shutdown_connections 2023-12-17T06:32:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:15.490+0000 7f38b425c640 1 -- 172.21.15.28:0/2277686288 wait complete. 2023-12-17T06:32:15.625 INFO:tasks.cephfs_test_runner:test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) ... FAIL 2023-12-17T06:32:15.626 INFO:tasks.cephfs_test_runner: 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner:====================================================================== 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner:FAIL: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/tests_from_xfstests_dev.py", line 12, in test_generic 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner: self.run_generic_tests() 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/xfstests_dev.py", line 407, in run_generic_tests 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner: return self.run_testdir('generic') 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/xfstests_dev.py", line 399, in run_testdir 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner: self.assertEqual(self.total_tests_failed, 0) 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner:AssertionError: 16 != 0 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner: 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner:Ran 1 test in 30844.445s 2023-12-17T06:32:15.627 INFO:tasks.cephfs_test_runner: 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner: 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner:====================================================================== 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner:FAIL: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/tests_from_xfstests_dev.py", line 12, in test_generic 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner: self.run_generic_tests() 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/xfstests_dev.py", line 407, in run_generic_tests 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner: return self.run_testdir('generic') 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/xfstests_dev.py", line 399, in run_testdir 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner: self.assertEqual(self.total_tests_failed, 0) 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner:AssertionError: 16 != 0 2023-12-17T06:32:15.628 INFO:tasks.cephfs_test_runner: 2023-12-17T06:32:15.629 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-12-17T06:32:15.707 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4f1ecda474a04a72955af7947be6cb51 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-12-17T06:32:15.709 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-12-17T06:32:15.719 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-12-17T06:32:15.740 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-12-17T06:32:15.740 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-17T06:32:15.740 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-17T06:32:15.808 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16339660k,nr_inodes=4084915,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19728 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275312k,nr_inodes=818828,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-17T06:32:15.808 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2023-12-17T06:32:15.808 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-17T06:32:15.910 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16339660k,nr_inodes=4084915,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19728 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275312k,nr_inodes=818828,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275316k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-17T06:32:15.911 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-17T06:32:15.911 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-17T06:32:15.918 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336396k,nr_inodes=4084099,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16145 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274660k,nr_inodes=818665,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-17T06:32:15.918 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-12-17T06:32:15.918 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-17T06:32:15.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336396k,nr_inodes=4084099,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16145 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274660k,nr_inodes=818665,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274664k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-17T06:32:15.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:15.994 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2023-12-17T06:32:16.131 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-12-17T06:32:16.132 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-17T06:32:16.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:16.132 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-17T06:32:16.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:16.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-17T06:32:16.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:16.281 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link delete ceph-brx 2023-12-17T06:32:16.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:16.544 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2023-12-17T06:32:16.562 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-17T06:32:16.562 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-17T06:32:16.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:16.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-17T06:32:16.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:16.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-17T06:32:16.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-17T06:32:16.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2023-12-17T06:32:17.140 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-17T06:32:17.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-17T06:32:17.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-17T06:32:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893f3cf640 1 Processor -- start 2023-12-17T06:32:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893f3cf640 1 -- start start 2023-12-17T06:32:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893f3cf640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f89380cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893f3cf640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89380ce840 con 0x7f893815d190 2023-12-17T06:32:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893f3cf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893815dd00 con 0x7f8938159490 2023-12-17T06:32:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893f3cf640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89380ce560 con 0x7f893815cdb0 2023-12-17T06:32:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893d16b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f89380cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893c96a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8938159490 0x7f89380b1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:48610/0 (socket says 172.21.15.28:48610) 2023-12-17T06:32:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893d16b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f89380cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37192/0 (socket says 172.21.15.28:37192) 2023-12-17T06:32:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893c96a640 1 -- 172.21.15.28:0/1732203019 learned_addr learned my addr 172.21.15.28:0/1732203019 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893d16b640 1 -- 172.21.15.28:0/1732203019 >> v1:172.21.15.122:6789/0 conn(0x7f8938159490 legacy=0x7f89380b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893d16b640 1 -- 172.21.15.28:0/1732203019 >> v1:172.21.15.28:6789/0 conn(0x7f893815d190 legacy=0x7f893815d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.550+0000 7f893d16b640 1 -- 172.21.15.28:0/1732203019 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8938070f80 con 0x7f893815cdb0 2023-12-17T06:32:17.564 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.554+0000 7f893d16b640 1 --2- 172.21.15.28:0/1732203019 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f89380cb690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f89280097f0 tx=0x7f892801c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae581b9b7b9b4d79 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:17.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.574+0000 7f892ffff640 1 -- 172.21.15.28:0/1732203019 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89280a7070 con 0x7f893815cdb0 2023-12-17T06:32:17.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.574+0000 7f892ffff640 1 -- 172.21.15.28:0/1732203019 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89280ab430 con 0x7f893815cdb0 2023-12-17T06:32:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.582+0000 7f892ffff640 1 -- 172.21.15.28:0/1732203019 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89280ab7c0 con 0x7f893815cdb0 2023-12-17T06:32:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.582+0000 7f893f3cf640 1 -- 172.21.15.28:0/1732203019 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 msgr2=0x7f89380cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.582+0000 7f893f3cf640 1 --2- 172.21.15.28:0/1732203019 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f89380cb690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f89280097f0 tx=0x7f892801c430 comp rx=0 tx=0).stop 2023-12-17T06:32:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.582+0000 7f893f3cf640 1 -- 172.21.15.28:0/1732203019 shutdown_connections 2023-12-17T06:32:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.582+0000 7f893f3cf640 1 --2- 172.21.15.28:0/1732203019 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f89380cb690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.582+0000 7f893f3cf640 1 -- 172.21.15.28:0/1732203019 >> 172.21.15.28:0/1732203019 conn(0x7f893806c2c0 msgr2=0x7f893806c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 -- 172.21.15.28:0/1732203019 shutdown_connections 2023-12-17T06:32:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 -- 172.21.15.28:0/1732203019 wait complete. 2023-12-17T06:32:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 Processor -- start 2023-12-17T06:32:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 -- start start 2023-12-17T06:32:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8938159490 0x7f89380adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f893814b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893815d190 0x7f8938214070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89381580a0 con 0x7f893815d190 2023-12-17T06:32:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893815af60 con 0x7f8938159490 2023-12-17T06:32:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893f3cf640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8938157b90 con 0x7f893815cdb0 2023-12-17T06:32:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893c96a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f893814b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893c96a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f893814b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37196/0 (socket says 172.21.15.28:37196) 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893c96a640 1 -- 172.21.15.28:0/3651215596 learned_addr learned my addr 172.21.15.28:0/3651215596 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893d16b640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8938159490 0x7f89380adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893d96c640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893815d190 0x7f8938214070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.590+0000 7f893c96a640 1 -- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8938159490 msgr2=0x7f89380adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.594+0000 7f893c96a640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8938159490 0x7f89380adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.594+0000 7f893c96a640 1 -- 172.21.15.28:0/3651215596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893815d190 msgr2=0x7f8938214070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.594+0000 7f893c96a640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893815d190 0x7f8938214070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.594+0000 7f893c96a640 1 -- 172.21.15.28:0/3651215596 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8928009420 con 0x7f893815cdb0 2023-12-17T06:32:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.594+0000 7f893d16b640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8938159490 0x7f89380adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.594+0000 7f893c96a640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f893814b3e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f893000cb10 tx=0x7f893000a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.598+0000 7f892dffb640 1 -- 172.21.15.28:0/3651215596 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893000ac70 con 0x7f893815cdb0 2023-12-17T06:32:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.598+0000 7f892dffb640 1 -- 172.21.15.28:0/3651215596 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893000add0 con 0x7f893815cdb0 2023-12-17T06:32:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.598+0000 7f892dffb640 1 -- 172.21.15.28:0/3651215596 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930012620 con 0x7f893815cdb0 2023-12-17T06:32:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.598+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89381554a0 con 0x7f893815cdb0 2023-12-17T06:32:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.598+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8938157dc0 con 0x7f893815cdb0 2023-12-17T06:32:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.598+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8938069440 con 0x7f893815cdb0 2023-12-17T06:32:17.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.614+0000 7f892dffb640 1 -- 172.21.15.28:0/3651215596 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f8930007070 con 0x7f893815cdb0 2023-12-17T06:32:17.619 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.614+0000 7f892dffb640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f89240666d0 0x7f8924068b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.614+0000 7f892dffb640 1 -- 172.21.15.28:0/3651215596 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8930083ed0 con 0x7f893815cdb0 2023-12-17T06:32:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.614+0000 7f893d16b640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f89240666d0 0x7f8924068b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.614+0000 7f892dffb640 1 -- 172.21.15.28:0/3651215596 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f89300842a0 con 0x7f893815cdb0 2023-12-17T06:32:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:17.614+0000 7f893d16b640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f89240666d0 0x7f8924068b90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f89280af000 tx=0x7f8928008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.046+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8918000d60 con 0x7f89240666d0 2023-12-17T06:32:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.050+0000 7f892dffb640 1 -- 172.21.15.28:0/3651215596 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48975 (secure 0 0 0) 0x7f8918000d60 con 0x7f89240666d0 2023-12-17T06:32:18.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:18.059 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-17T06:32:18.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f89240666d0 msgr2=0x7f8924068b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:18.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f89240666d0 0x7f8924068b90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f89280af000 tx=0x7f8928008040 comp rx=0 tx=0).stop 2023-12-17T06:32:18.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 msgr2=0x7f893814b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:18.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f893814b3e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f893000cb10 tx=0x7f893000a060 comp rx=0 tx=0).stop 2023-12-17T06:32:18.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 shutdown_connections 2023-12-17T06:32:18.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f89240666d0 0x7f8924068b90 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893815d190 0x7f8938214070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f893815cdb0 0x7f893814b3e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 --2- 172.21.15.28:0/3651215596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8938159490 0x7f89380adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.062+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 >> 172.21.15.28:0/3651215596 conn(0x7f893806c2c0 msgr2=0x7f8938158230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:18.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.070+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 shutdown_connections 2023-12-17T06:32:18.075 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.070+0000 7f893f3cf640 1 -- 172.21.15.28:0/3651215596 wait complete. 2023-12-17T06:32:18.111 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":15470,"stamp":"2023-12-17T06:32:16.770917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":56315780,"kb_used_data":43562788,"kb_used_omap":3694,"kb_used_meta":12749201,"kb_avail":698658940,"statfs":{"total":773094113280,"available":715426754560,"internally_reserved":0,"allocated":44608294912,"data_stored":44510649656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3783506,"internal_metadata":13055181998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-43519093256,"num_objects":-35401,"num_object_clones":0,"num_object_copies":-70802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-35401,"num_whiteouts":0,"num_read":-1890626,"num_read_kb":-132487568,"num_write":-7495217,"num_write_kb":-162617007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2362,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.017793"},"pg_stats":[{"pgid":"1.0","version":"36'161","reported_seq":484,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-17T06:32:09.754483+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-17T06:32:09.754483+0000","last_peered":"2023-12-17T06:32:09.754483+0000","last_clean":"2023-12-17T06:32:09.754483+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-17T06:32:09.754483+0000","last_undegraded":"2023-12-17T06:32:09.754483+0000","last_fullsized":"2023-12-17T06:32:09.754483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719482916,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7340040,"kb_used_data":5844604,"kb_used_omap":20,"kb_used_meta":1495403,"kb_avail":87031800,"statfs":{"total":96636764160,"available":89120563200,"internally_reserved":0,"allocated":5984874496,"data_stored":5972031162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20951,"internal_metadata":1531293225},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.9750000000000001,"5min":1.1080000000000001,"15min":0.88500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":15.872,"5min":15.872,"15min":15.872},"last":2.4199999999999999},{"interface":"front","average":{"1min":2.355,"5min":1.238,"15min":0.96999999999999997},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":22.158000000000001,"5min":22.158000000000001,"15min":22.158000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.97099999999999997,"15min":0.82199999999999995},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":8.2279999999999998,"5min":8.2279999999999998,"15min":11.856},"last":2.4689999999999999},{"interface":"front","average":{"1min":2.6429999999999998,"5min":1.4390000000000001,"15min":1.1599999999999999},"min":{"1min":0.30199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":19.102,"5min":19.102,"15min":19.102},"last":0.49399999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.327,"15min":1.196},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":19.332000000000001,"5min":19.332000000000001,"15min":19.332000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.593,"5min":1.518,"15min":1.2969999999999999},"min":{"1min":0.33600000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":15.898,"5min":15.898,"15min":15.898},"last":2.3370000000000002}]},{"osd":3,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":3.137,"5min":1.635,"15min":1.365},"min":{"1min":0.35399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":24.094999999999999,"5min":24.094999999999999,"15min":24.094999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.3330000000000002,"5min":1.5529999999999999,"15min":1.26},"min":{"1min":0.39300000000000002,"5min":0.32500000000000001,"15min":0.28999999999999998},"max":{"1min":15.837,"5min":15.837,"15min":15.837},"last":2.3730000000000002}]},{"osd":4,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.5089999999999999,"5min":1.141,"15min":0.86399999999999999},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27100000000000002},"max":{"1min":16.614000000000001,"5min":16.614000000000001,"15min":16.614000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.2989999999999999,"5min":0.83599999999999997,"15min":0.78600000000000003},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":8.2699999999999996,"5min":8.2699999999999996,"15min":8.2699999999999996},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":0.93100000000000005,"15min":0.88900000000000001},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":8.1969999999999992,"5min":8.1969999999999992,"15min":11.741},"last":2.2440000000000002},{"interface":"front","average":{"1min":2.8210000000000002,"5min":1.2929999999999999,"15min":1.024},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":27.901,"5min":27.901,"15min":27.901},"last":0.83299999999999996}]},{"osd":6,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.0860000000000001,"15min":0.90200000000000002},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":0.995},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.0660000000000001,"15min":0.95099999999999996},"min":{"1min":0.36299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.946,"5min":11.946,"15min":11.946},"last":0.56200000000000006}]}]},{"osd":6,"up_from":16,"seq":68719482915,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5833092,"kb_used_data":4098488,"kb_used_omap":981,"kb_used_meta":1733610,"kb_avail":88538748,"statfs":{"total":96636764160,"available":90663677952,"internally_reserved":0,"allocated":4196851712,"data_stored":4187815407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1005312,"internal_metadata":1775216896},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2879999999999998,"5min":1.2549999999999999,"15min":0.88900000000000001},"min":{"1min":0.26600000000000001,"5min":0.24399999999999999,"15min":0.22500000000000001},"max":{"1min":38.720999999999997,"5min":38.720999999999997,"15min":38.720999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.0669999999999999,"15min":0.90200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26500000000000001,"15min":0.25},"max":{"1min":5.3920000000000003,"5min":8.0779999999999994,"15min":8.0779999999999994},"last":1.093}]},{"osd":1,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.1850000000000001,"15min":1.1990000000000001},"min":{"1min":0.26800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.2619999999999996,"5min":5.077,"15min":8.9410000000000007},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.639,"5min":1.1419999999999999,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":4.7590000000000003,"5min":8.798,"15min":8.798},"last":1.0149999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.3819999999999999,"15min":1.167},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.5819999999999999,"5min":8.9090000000000007,"15min":8.9090000000000007},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.496,"5min":1.425,"15min":1.236},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.29599999999999999},"max":{"1min":4.9790000000000001,"5min":8.8179999999999996,"15min":10.401999999999999},"last":1.05}]},{"osd":3,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2029999999999998,"5min":1.8009999999999999,"15min":1.369},"min":{"1min":0.38700000000000001,"5min":0.31,"15min":0.31},"max":{"1min":16.004000000000001,"5min":16.004000000000001,"15min":16.004000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.339,"5min":1.1120000000000001,"15min":1.2470000000000001},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.32000000000000001},"max":{"1min":4.867,"5min":8.2040000000000006,"15min":8.2040000000000006},"last":0.94999999999999996}]},{"osd":4,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.044,"15min":0.89000000000000001},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.25900000000000001},"max":{"1min":20.827000000000002,"5min":20.827000000000002,"15min":20.827000000000002},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0169999999999999,"15min":0.82899999999999996},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":10.022,"5min":10.022,"15min":10.022},"last":1.117}]},{"osd":5,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.8079999999999998,"5min":1.448,"15min":1.073},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":41.658000000000001,"5min":41.658000000000001,"15min":41.658000000000001},"last":1.194},{"interface":"front","average":{"1min":3.9060000000000001,"5min":1.423,"15min":1.0580000000000001},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.29199999999999998},"max":{"1min":41.634999999999998,"5min":41.634999999999998,"15min":41.634999999999998},"last":1.25}]},{"osd":7,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.012,"15min":0.93700000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":4.7960000000000003,"5min":5.2859999999999996,"15min":5.625},"last":1.286},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.82499999999999996,"15min":0.874},"min":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.318},"max":{"1min":3.835,"5min":4.5359999999999996,"15min":7.0810000000000004},"last":1.175}]}]},{"osd":5,"up_from":16,"seq":68719482916,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7277404,"kb_used_data":5634448,"kb_used_omap":463,"kb_used_meta":1642480,"kb_avail":87094436,"statfs":{"total":96636764160,"available":89184702464,"internally_reserved":0,"allocated":5769674752,"data_stored":5756769543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475086,"internal_metadata":1681899570},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.66300000000000003,"15min":0.73799999999999999},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.23200000000000001},"max":{"1min":5.2619999999999996,"5min":5.2619999999999996,"15min":6.2859999999999996},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.90700000000000003,"15min":0.84299999999999997},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":3.6600000000000001,"5min":5.5810000000000004,"15min":6.4829999999999997},"last":0.67700000000000005}]},{"osd":1,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.0549999999999999,"15min":1.018},"min":{"1min":0.27700000000000002,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":5.8109999999999999,"5min":9.2159999999999993,"15min":9.2159999999999993},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.92500000000000004,"15min":0.875},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":8.6959999999999997},"last":0.76700000000000002}]},{"osd":2,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.92700000000000005,"15min":0.90200000000000002},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28699999999999998},"max":{"1min":5.0919999999999996,"5min":5.4290000000000003,"15min":6.3460000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.196,"15min":1.145},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":5.8810000000000002,"5min":5.8810000000000002,"15min":6.3540000000000001},"last":0.90700000000000003}]},{"osd":3,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":1.181,"15min":1.1160000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.307},"max":{"1min":20.481000000000002,"5min":20.481000000000002,"15min":20.481000000000002},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.25,"15min":1.1870000000000001},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.29099999999999998},"max":{"1min":5.8520000000000003,"5min":5.8520000000000003,"15min":6.444},"last":1.024}]},{"osd":4,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.66900000000000004,"15min":0.68899999999999995},"min":{"1min":0.314,"5min":0.249,"15min":0.249},"max":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":6.2149999999999999},"last":1.125},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79700000000000004,"15min":0.78400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.3490000000000002,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.2749999999999999}]},{"osd":6,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":1.1319999999999999,"15min":0.98699999999999999},"min":{"1min":0.38700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":27.823,"5min":27.823,"15min":27.823},"last":1.0960000000000001},{"interface":"front","average":{"1min":2.5529999999999999,"5min":1.0109999999999999,"15min":0.86199999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":33.023000000000003,"5min":33.023000000000003,"15min":33.023000000000003},"last":1.3280000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.5230000000000001,"5min":1.4139999999999999,"15min":1.0029999999999999},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":42.939,"5min":42.939,"15min":42.939},"last":1.3580000000000001},{"interface":"front","average":{"1min":2.234,"5min":1.0629999999999999,"15min":1.0209999999999999},"min":{"1min":0.44600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":18.196999999999999,"5min":18.196999999999999,"15min":18.196999999999999},"last":1.222}]}]},{"osd":4,"up_from":16,"seq":68719482914,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8047568,"kb_used_data":6360644,"kb_used_omap":362,"kb_used_meta":1686549,"kb_avail":86324272,"statfs":{"total":96636764160,"available":88396054528,"internally_reserved":0,"allocated":6513299456,"data_stored":6498755697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":371446,"internal_metadata":1727026442},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.57599999999999996,"15min":0.69199999999999995},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.246},"max":{"1min":1.2909999999999999,"5min":4.8220000000000001,"15min":6.8559999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.244,"5min":0.94799999999999995,"15min":0.91400000000000003},"min":{"1min":0.309,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.5,"5min":7.1079999999999997,"15min":8.9420000000000002},"last":0.81799999999999995}]},{"osd":1,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.0249999999999999,"15min":1.0489999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.251},"max":{"1min":6.5339999999999998,"5min":7.3399999999999999,"15min":7.3399999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81100000000000005,"15min":1.161},"min":{"1min":0.28699999999999998,"5min":0.27300000000000002,"15min":0.25900000000000001},"max":{"1min":3.6709999999999998,"5min":7.1849999999999996,"15min":27.398},"last":0.35899999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.95899999999999996,"15min":0.98499999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":5.2569999999999997,"5min":7.9189999999999996,"15min":7.9189999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.964,"5min":1.5229999999999999,"15min":1.4219999999999999},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.29299999999999998},"max":{"1min":6.641,"5min":10.103,"15min":10.103},"last":0.74399999999999999}]},{"osd":3,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.234,"15min":1.333},"min":{"1min":0.33700000000000002,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":5.1349999999999998,"5min":11.417999999999999,"15min":12.627000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.0629999999999999,"15min":1.2110000000000001},"min":{"1min":0.32400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":4.2089999999999996,"5min":8.1259999999999994,"15min":10.43},"last":0.39400000000000002}]},{"osd":5,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81599999999999995,"15min":0.78000000000000003},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.4660000000000002,"5min":6.4660000000000002,"15min":8.7249999999999996},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.61199999999999999,"15min":0.73899999999999999},"min":{"1min":0.33700000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.105,"5min":4.2629999999999999,"15min":6.6559999999999997},"last":0.53700000000000003}]},{"osd":6,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.72199999999999998,"15min":0.80100000000000005},"min":{"1min":0.34699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.1990000000000001,"5min":7.5999999999999996,"15min":7.5999999999999996},"last":0.44},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.004,"15min":0.91000000000000003},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.34000000000000002},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"last":0.87}]},{"osd":7,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.74199999999999999,"15min":0.84499999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":6.1219999999999999,"5min":6.1219999999999999,"15min":6.6879999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.88400000000000001,"15min":0.91300000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80000000000000004,"5min":5.2919999999999998,"15min":6.9379999999999997},"last":1.246}]}]},{"osd":3,"up_from":15,"seq":64424515608,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8242848,"kb_used_data":6540564,"kb_used_omap":132,"kb_used_meta":1702139,"kb_avail":86128992,"statfs":{"total":96636764160,"available":88196087808,"internally_reserved":0,"allocated":6697537536,"data_stored":6682960825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136113,"internal_metadata":1742990415},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":1.1759999999999999,"15min":1.1319999999999999},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.20000000000000001},"max":{"1min":21.396999999999998,"5min":21.396999999999998,"15min":21.396999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.393,"5min":0.93799999999999994,"15min":0.92000000000000004},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":12.132,"5min":12.132,"15min":12.132},"last":0.749}]},{"osd":1,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":1.339,"15min":1.1479999999999999},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.215},"max":{"1min":29.577000000000002,"5min":29.577000000000002,"15min":29.577000000000002},"last":0.879},{"interface":"front","average":{"1min":2.0870000000000002,"5min":1.4950000000000001,"15min":1.373},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":12.877000000000001,"5min":12.877000000000001,"15min":12.877000000000001},"last":0.70199999999999996}]},{"osd":2,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":1.3200000000000001,"15min":1.395},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.253},"max":{"1min":12.920999999999999,"5min":12.920999999999999,"15min":12.920999999999999},"last":1.109},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.3380000000000001,"15min":1.177},"min":{"1min":0.34999999999999998,"5min":0.29799999999999999,"15min":0.27500000000000002},"max":{"1min":17.669,"5min":17.669,"15min":17.669},"last":1.1559999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.6339999999999999,"5min":1.488,"15min":1.2769999999999999},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":13.095000000000001,"5min":13.228,"15min":13.228},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.161,"15min":1.0860000000000001},"min":{"1min":0.35899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":12.166,"5min":12.166,"15min":12.166},"last":1.198}]},{"osd":5,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.3889999999999998,"5min":1.4710000000000001,"15min":1.234},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":29.622,"5min":29.622,"15min":29.622},"last":1.044},{"interface":"front","average":{"1min":3.5619999999999998,"5min":1.6699999999999999,"15min":1.409},"min":{"1min":0.38100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":21.038,"5min":21.038,"15min":21.038},"last":1.244}]},{"osd":6,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.1539999999999999,"15min":1.105},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.34999999999999998},"max":{"1min":4.0380000000000003,"5min":5.5220000000000002,"15min":10.683},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.786,"5min":1.296,"15min":1.1779999999999999},"min":{"1min":0.47299999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":15.619999999999999,"5min":15.619999999999999,"15min":15.619999999999999},"last":1.286}]},{"osd":7,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.2989999999999999,"5min":1.71,"15min":1.4810000000000001},"min":{"1min":0.42099999999999999,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":18.16,"5min":18.16,"15min":18.16},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.643,"15min":1.288},"min":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.41799999999999998},"max":{"1min":10.282999999999999,"5min":22.334,"15min":22.334},"last":1.613}]}]},{"osd":1,"up_from":15,"seq":64424515612,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8149736,"kb_used_data":6316060,"kb_used_omap":997,"kb_used_meta":1832666,"kb_avail":86222104,"statfs":{"total":96636764160,"available":88291434496,"internally_reserved":0,"allocated":6467645440,"data_stored":6453248942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021567,"internal_metadata":1876650369},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.81499999999999995,"15min":0.89000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.5490000000000004,"5min":9.8010000000000002,"15min":9.8010000000000002},"last":0.434},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.089,"15min":1.196},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.214},"max":{"1min":30.109999999999999,"5min":30.109999999999999,"15min":30.109999999999999},"last":0.26900000000000002}]},{"osd":2,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5129999999999999,"5min":1.2869999999999999,"15min":1.202},"min":{"1min":0.30499999999999999,"5min":0.23300000000000001,"15min":0.224},"max":{"1min":25.259,"5min":25.259,"15min":25.259},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.873,"15min":0.998},"min":{"1min":0.29199999999999998,"5min":0.245,"15min":0.22500000000000001},"max":{"1min":5.4580000000000002,"5min":9.7720000000000002,"15min":9.7720000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.8199999999999998,"5min":1.218,"15min":1.2929999999999999},"min":{"1min":0.314,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":30.460999999999999,"5min":30.460999999999999,"15min":30.460999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.8120000000000001,"5min":1.4039999999999999,"15min":1.3080000000000001},"min":{"1min":0.311,"5min":0.28999999999999998,"15min":0.25},"max":{"1min":6.6210000000000004,"5min":8.0980000000000008,"15min":9.6460000000000008},"last":3.3690000000000002}]},{"osd":4,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1519999999999999,"15min":0.96899999999999997},"min":{"1min":0.34999999999999998,"5min":0.31,"15min":0.27300000000000002},"max":{"1min":5.5190000000000001,"5min":9.6669999999999998,"15min":10.122},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.0780000000000001,"15min":1.0600000000000001},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":0.63800000000000001}]},{"osd":5,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.127,"15min":0.99199999999999999},"min":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.315},"max":{"1min":5.492,"5min":9.7110000000000003,"15min":9.7110000000000003},"last":2.4569999999999999},{"interface":"front","average":{"1min":2.851,"5min":1.218,"15min":1.2490000000000001},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":30.277999999999999,"5min":30.277999999999999,"15min":30.277999999999999},"last":2.3199999999999998}]},{"osd":6,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1830000000000001,"15min":1.1819999999999999},"min":{"1min":0.46700000000000003,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":4.8200000000000003,"5min":6.3979999999999997,"15min":8.8840000000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.5109999999999999,"15min":1.1779999999999999},"min":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38700000000000001},"max":{"1min":19.016999999999999,"5min":19.016999999999999,"15min":19.016999999999999},"last":0.61699999999999999}]},{"osd":7,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.9620000000000002,"5min":1.401,"15min":1.284},"min":{"1min":0.52600000000000002,"5min":0.40699999999999997,"15min":0.378},"max":{"1min":30.242999999999999,"5min":30.242999999999999,"15min":30.242999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.659,"5min":1.2869999999999999,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":4.9240000000000004,"5min":6.46,"15min":8.9589999999999996},"last":0.71199999999999997}]}]},{"osd":0,"up_from":15,"seq":64424515615,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5625040,"kb_used_data":4244484,"kb_used_omap":422,"kb_used_meta":1380121,"kb_avail":88746800,"statfs":{"total":96636764160,"available":90876723200,"internally_reserved":0,"allocated":4346351616,"data_stored":4336938095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":432402,"internal_metadata":1413244654},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":0.88400000000000001,"15min":0.97199999999999998},"min":{"1min":0.20200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":8.3789999999999996,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.96199999999999997,"15min":1.181},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1079999999999997,"5min":5.1470000000000002,"15min":8.1039999999999992},"last":0.247}]},{"osd":2,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.159,"15min":1.194},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.2759999999999998,"5min":7.2759999999999998,"15min":10.146000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.2130000000000001,"15min":1.2889999999999999},"min":{"1min":0.26600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.4770000000000003,"5min":13.676,"15min":13.676},"last":0.39200000000000002}]},{"osd":3,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.2929999999999999,"15min":1.2749999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.25},"max":{"1min":2.5680000000000001,"5min":13.721,"15min":13.721},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.0840000000000001,"15min":1.1870000000000001},"min":{"1min":0.37,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":6.2089999999999996,"5min":8.0570000000000004,"15min":10.037000000000001},"last":1.5880000000000001}]},{"osd":4,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.145,"15min":1.1919999999999999},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.0579999999999998,"5min":13.755000000000001,"15min":13.755000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.0389999999999999,"15min":0.93200000000000005},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.29199999999999998},"max":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":7.3540000000000001},"last":0.30299999999999999}]},{"osd":5,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":0.95499999999999996,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.32700000000000001},"max":{"1min":6.5090000000000003,"5min":6.5090000000000003,"15min":7.4349999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.0089999999999999,"15min":1.149},"min":{"1min":0.379,"5min":0.31,"15min":0.31},"max":{"1min":4.1589999999999998,"5min":5.101,"15min":8.1820000000000004},"last":0.58999999999999997}]},{"osd":6,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.25,"15min":1.3069999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.34300000000000003},"max":{"1min":6.202,"5min":13.819000000000001,"15min":13.819000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.1499999999999999,"15min":1.087},"min":{"1min":0.38,"5min":0.38,"15min":0.378},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":0.46300000000000002}]},{"osd":7,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.087,"15min":1.2030000000000001},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":4.2119999999999997,"5min":5.1760000000000002,"15min":8.2110000000000003},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.266,"15min":1.4059999999999999},"min":{"1min":0.39500000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.242,"5min":13.786,"15min":13.786},"last":0.44}]}]},{"osd":2,"up_from":14,"seq":60129548316,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5800052,"kb_used_data":4523496,"kb_used_omap":313,"kb_used_meta":1276230,"kb_avail":88571788,"statfs":{"total":96636764160,"available":90697510912,"internally_reserved":0,"allocated":4632059904,"data_stored":4622129985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":320629,"internal_metadata":1306860427},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":0.96099999999999997,"15min":1.0049999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.8399999999999999,"5min":7.5919999999999996,"15min":8.1500000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3120000000000001,"15min":1.1439999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":10.301,"5min":10.301,"15min":10.301},"last":0.59499999999999997}]},{"osd":1,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0329999999999999,"15min":1.103},"min":{"1min":0.26600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":4.9039999999999999,"5min":6.8010000000000002,"15min":7.7320000000000002},"last":0.438},{"interface":"front","average":{"1min":1.181,"5min":1.333,"15min":1.27},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":10.345000000000001,"5min":10.345000000000001,"15min":10.345000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":1.446,"15min":1.4199999999999999},"min":{"1min":0.26700000000000002,"5min":0.24299999999999999,"15min":0.23999999999999999},"max":{"1min":10.706,"5min":10.706,"15min":10.706},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.042,"15min":1.1990000000000001},"min":{"1min":0.36699999999999999,"5min":0.29699999999999999,"15min":0.26100000000000001},"max":{"1min":4.9340000000000002,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.58899999999999997}]},{"osd":4,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.999,"15min":1.0920000000000001},"min":{"1min":0.374,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":4.7990000000000004,"5min":6.1870000000000003,"15min":8.0860000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.29,"15min":1.159},"min":{"1min":0.375,"5min":0.314,"15min":0.314},"max":{"1min":4.1630000000000003,"5min":9.4399999999999995,"15min":9.4399999999999995},"last":0.66000000000000003}]},{"osd":5,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.383,"15min":1.2649999999999999},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.9779999999999998,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.341,"15min":1.3759999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":7.8929999999999998,"5min":7.8929999999999998,"15min":10.366},"last":0.40899999999999997}]},{"osd":6,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":1.427,"15min":1.431},"min":{"1min":0.45900000000000002,"5min":0.39600000000000002,"15min":0.376},"max":{"1min":7.9770000000000003,"5min":7.9770000000000003,"15min":10.412000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.075,"15min":1.196},"min":{"1min":0.50800000000000001,"5min":0.36499999999999999,"15min":0.35999999999999999},"max":{"1min":4.8719999999999999,"5min":7.8959999999999999,"15min":8.0350000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.444,"15min":1.2909999999999999},"min":{"1min":0.44400000000000001,"5min":0.40100000000000002,"15min":0.37},"max":{"1min":5.8949999999999996,"5min":9.5489999999999995,"15min":9.5489999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":2.4449999999999998,"5min":1.456,"15min":1.4670000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":7.9429999999999996,"5min":7.9429999999999996,"15min":10.446},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-17T06:32:18.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-17T06:32:18.487 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c8909640 1 Processor -- start 2023-12-17T06:32:18.487 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c8909640 1 -- start start 2023-12-17T06:32:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c8909640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f84c4158f50 0x7f84c4159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c8909640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e210 0x7f84c40b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c8909640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c40b1c20 con 0x7f84c406e650 2023-12-17T06:32:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c8909640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c406eb60 con 0x7f84c4158f50 2023-12-17T06:32:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c8909640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4159910 con 0x7f84c406e210 2023-12-17T06:32:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c259c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e210 0x7f84c40b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c2d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f84c4158f50 0x7f84c4159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c259c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e210 0x7f84c40b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37204/0 (socket says 172.21.15.28:37204) 2023-12-17T06:32:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c259c640 1 -- 172.21.15.28:0/378772874 learned_addr learned my addr 172.21.15.28:0/378772874 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c2d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f84c4158f50 0x7f84c4159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36622/0 (socket says 172.21.15.28:36622) 2023-12-17T06:32:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c259c640 1 -- 172.21.15.28:0/378772874 >> v2:172.21.15.28:3300/0 conn(0x7f84c4158f50 msgr2=0x7f84c4159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c259c640 1 --2- 172.21.15.28:0/378772874 >> v2:172.21.15.28:3300/0 conn(0x7f84c4158f50 0x7f84c4159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c259c640 1 -- 172.21.15.28:0/378772874 >> v1:172.21.15.28:6789/0 conn(0x7f84c406e650 legacy=0x7f84c40cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.482+0000 7f84c259c640 1 -- 172.21.15.28:0/378772874 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c4159bf0 con 0x7f84c406e210 2023-12-17T06:32:18.492 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.486+0000 7f84c259c640 1 --2- 172.21.15.28:0/378772874 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e210 0x7f84c40b3590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009e50 tx=0x7f84bc01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eff1df85214f73b4 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c1d9b640 1 -- 172.21.15.28:0/378772874 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0a56c0 con 0x7f84c406e210 2023-12-17T06:32:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c1d9b640 1 -- 172.21.15.28:0/378772874 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc0a5820 con 0x7f84c406e210 2023-12-17T06:32:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 -- 172.21.15.28:0/378772874 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e210 msgr2=0x7f84c40b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 --2- 172.21.15.28:0/378772874 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e210 0x7f84c40b3590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009e50 tx=0x7f84bc01c910 comp rx=0 tx=0).stop 2023-12-17T06:32:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 -- 172.21.15.28:0/378772874 shutdown_connections 2023-12-17T06:32:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 --2- 172.21.15.28:0/378772874 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e210 0x7f84c40b3590 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 --2- 172.21.15.28:0/378772874 >> v2:172.21.15.28:3300/0 conn(0x7f84c4158f50 0x7f84c4159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 -- 172.21.15.28:0/378772874 >> 172.21.15.28:0/378772874 conn(0x7f84c40b0920 msgr2=0x7f84c40b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 -- 172.21.15.28:0/378772874 shutdown_connections 2023-12-17T06:32:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 -- 172.21.15.28:0/378772874 wait complete. 2023-12-17T06:32:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 Processor -- start 2023-12-17T06:32:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.494+0000 7f84c8909640 1 -- start start 2023-12-17T06:32:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c8909640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84c406e210 0x7f84c4156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c8909640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e650 0x7f84c41446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c8909640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c4158f50 0x7f84c4164f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c8909640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4162430 con 0x7f84c406e210 2023-12-17T06:32:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c8909640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c414a690 con 0x7f84c4158f50 2023-12-17T06:32:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c8909640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4156eb0 con 0x7f84c406e650 2023-12-17T06:32:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c259c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e650 0x7f84c41446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c259c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e650 0x7f84c41446a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37212/0 (socket says 172.21.15.28:37212) 2023-12-17T06:32:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c259c640 1 -- 172.21.15.28:0/1545076087 learned_addr learned my addr 172.21.15.28:0/1545076087 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c359e640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c4158f50 0x7f84c4164f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c2d9d640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84c406e210 0x7f84c4156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c359e640 1 -- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e650 msgr2=0x7f84c41446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c359e640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e650 0x7f84c41446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c359e640 1 -- 172.21.15.28:0/1545076087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84c406e210 msgr2=0x7f84c4156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c359e640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84c406e210 0x7f84c4156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c359e640 1 -- 172.21.15.28:0/1545076087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc009b30 con 0x7f84c4158f50 2023-12-17T06:32:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.498+0000 7f84c359e640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c4158f50 0x7f84c4164f50 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f84b800c3e0 tx=0x7f84b800c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.510+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1545076087 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8011790 con 0x7f84c4158f50 2023-12-17T06:32:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.510+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1545076087 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b80118f0 con 0x7f84c4158f50 2023-12-17T06:32:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.510+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c4149a80 con 0x7f84c4158f50 2023-12-17T06:32:18.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.510+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c414a040 con 0x7f84c4158f50 2023-12-17T06:32:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.510+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1545076087 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8011ce0 con 0x7f84c4158f50 2023-12-17T06:32:18.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.510+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c40cd800 con 0x7f84c4158f50 2023-12-17T06:32:18.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.518+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1545076087 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f84b8005080 con 0x7f84c4158f50 2023-12-17T06:32:18.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.518+0000 7f84ab7fe640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f84b00665c0 0x7f84b0068a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:18.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.518+0000 7f84c2d9d640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f84b00665c0 0x7f84b0068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:18.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.518+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1545076087 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f84b80835a0 con 0x7f84c4158f50 2023-12-17T06:32:18.524 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.518+0000 7f84c2d9d640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f84b00665c0 0x7f84b0068a80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f84b40099d0 tx=0x7f84b4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.522+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1545076087 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f84b8050d90 con 0x7f84c4158f50 2023-12-17T06:32:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.970+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84c40b3ee0 con 0x7f84b00665c0 2023-12-17T06:32:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:18.974+0000 7f84ab7fe640 1 -- 172.21.15.28:0/1545076087 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48990 (secure 0 0 0) 0x7f84b40118e0 con 0x7f84b00665c0 2023-12-17T06:32:18.983 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-17T06:32:19.011 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f84b00665c0 msgr2=0x7f84b0068a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f84b00665c0 0x7f84b0068a80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f84b40099d0 tx=0x7f84b4008040 comp rx=0 tx=0).stop 2023-12-17T06:32:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c4158f50 msgr2=0x7f84c4164f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c4158f50 0x7f84c4164f50 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f84b800c3e0 tx=0x7f84b800c8a0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 shutdown_connections 2023-12-17T06:32:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f84b00665c0 0x7f84b0068a80 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84c4158f50 0x7f84c4164f50 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f84c406e650 0x7f84c41446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 --2- 172.21.15.28:0/1545076087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84c406e210 0x7f84c4156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 >> 172.21.15.28:0/1545076087 conn(0x7f84c40b0920 msgr2=0x7f84c40b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 shutdown_connections 2023-12-17T06:32:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.006+0000 7f84c8909640 1 -- 172.21.15.28:0/1545076087 wait complete. 2023-12-17T06:32:19.036 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":15471,"stamp":"2023-12-17T06:32:18.771238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":46931448,"kb_used_data":34142368,"kb_used_omap":3694,"kb_used_meta":12785297,"kb_avail":708043272,"statfs":{"total":773094113280,"available":725036310528,"internally_reserved":0,"allocated":34961784832,"data_stored":34884853364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3783506,"internal_metadata":13092144302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":250,"apply_latency_ms":250,"commit_latency_ns":250000000,"apply_latency_ns":250000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-43516366526,"num_objects":-35384,"num_object_clones":0,"num_object_copies":-70768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-35384,"num_whiteouts":0,"num_read":-1890626,"num_read_kb":-132487568,"num_write":-7495234,"num_write_kb":-162617007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2360,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001630"},"pg_stats":[{"pgid":"1.0","version":"36'161","reported_seq":484,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-17T06:32:09.754483+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-17T06:32:09.754483+0000","last_peered":"2023-12-17T06:32:09.754483+0000","last_clean":"2023-12-17T06:32:09.754483+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-17T06:32:09.754483+0000","last_undegraded":"2023-12-17T06:32:09.754483+0000","last_fullsized":"2023-12-17T06:32:09.754483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719482916,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7340040,"kb_used_data":5844604,"kb_used_omap":20,"kb_used_meta":1495403,"kb_avail":87031800,"statfs":{"total":96636764160,"available":89120563200,"internally_reserved":0,"allocated":5984874496,"data_stored":5972031162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20951,"internal_metadata":1531293225},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.9750000000000001,"5min":1.1080000000000001,"15min":0.88500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":15.872,"5min":15.872,"15min":15.872},"last":2.4199999999999999},{"interface":"front","average":{"1min":2.355,"5min":1.238,"15min":0.96999999999999997},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":22.158000000000001,"5min":22.158000000000001,"15min":22.158000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.97099999999999997,"15min":0.82199999999999995},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":8.2279999999999998,"5min":8.2279999999999998,"15min":11.856},"last":2.4689999999999999},{"interface":"front","average":{"1min":2.6429999999999998,"5min":1.4390000000000001,"15min":1.1599999999999999},"min":{"1min":0.30199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":19.102,"5min":19.102,"15min":19.102},"last":0.49399999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.327,"15min":1.196},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":19.332000000000001,"5min":19.332000000000001,"15min":19.332000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.593,"5min":1.518,"15min":1.2969999999999999},"min":{"1min":0.33600000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":15.898,"5min":15.898,"15min":15.898},"last":2.3370000000000002}]},{"osd":3,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":3.137,"5min":1.635,"15min":1.365},"min":{"1min":0.35399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":24.094999999999999,"5min":24.094999999999999,"15min":24.094999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.3330000000000002,"5min":1.5529999999999999,"15min":1.26},"min":{"1min":0.39300000000000002,"5min":0.32500000000000001,"15min":0.28999999999999998},"max":{"1min":15.837,"5min":15.837,"15min":15.837},"last":2.3730000000000002}]},{"osd":4,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.5089999999999999,"5min":1.141,"15min":0.86399999999999999},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27100000000000002},"max":{"1min":16.614000000000001,"5min":16.614000000000001,"15min":16.614000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.2989999999999999,"5min":0.83599999999999997,"15min":0.78600000000000003},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":8.2699999999999996,"5min":8.2699999999999996,"15min":8.2699999999999996},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":0.93100000000000005,"15min":0.88900000000000001},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":8.1969999999999992,"5min":8.1969999999999992,"15min":11.741},"last":2.2440000000000002},{"interface":"front","average":{"1min":2.8210000000000002,"5min":1.2929999999999999,"15min":1.024},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":27.901,"5min":27.901,"15min":27.901},"last":0.83299999999999996}]},{"osd":6,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.0860000000000001,"15min":0.90200000000000002},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":0.995},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.0660000000000001,"15min":0.95099999999999996},"min":{"1min":0.36299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.946,"5min":11.946,"15min":11.946},"last":0.56200000000000006}]}]},{"osd":6,"up_from":16,"seq":68719482916,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1939272,"kb_used_data":168576,"kb_used_omap":981,"kb_used_meta":1769706,"kb_avail":92432568,"statfs":{"total":96636764160,"available":94650949632,"internally_reserved":0,"allocated":172621824,"data_stored":171894573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1005312,"internal_metadata":1812179200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":96,"apply_latency_ms":96,"commit_latency_ns":96000000,"apply_latency_ns":96000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2879999999999998,"5min":1.2549999999999999,"15min":0.88900000000000001},"min":{"1min":0.26600000000000001,"5min":0.24399999999999999,"15min":0.22500000000000001},"max":{"1min":38.720999999999997,"5min":38.720999999999997,"15min":38.720999999999997},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.0669999999999999,"15min":0.90200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26500000000000001,"15min":0.25},"max":{"1min":5.3920000000000003,"5min":8.0779999999999994,"15min":8.0779999999999994},"last":0.41299999999999998}]},{"osd":1,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.1850000000000001,"15min":1.1990000000000001},"min":{"1min":0.26800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.2619999999999996,"5min":5.077,"15min":8.9410000000000007},"last":1.083},{"interface":"front","average":{"1min":1.639,"5min":1.1419999999999999,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":4.7590000000000003,"5min":8.798,"15min":8.798},"last":0.998}]},{"osd":2,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.3819999999999999,"15min":1.167},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.5819999999999999,"5min":8.9090000000000007,"15min":8.9090000000000007},"last":11.710000000000001},{"interface":"front","average":{"1min":1.496,"5min":1.425,"15min":1.236},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.29599999999999999},"max":{"1min":4.9790000000000001,"5min":8.8179999999999996,"15min":10.401999999999999},"last":11.788}]},{"osd":3,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2029999999999998,"5min":1.8009999999999999,"15min":1.369},"min":{"1min":0.38700000000000001,"5min":0.31,"15min":0.31},"max":{"1min":16.004000000000001,"5min":16.004000000000001,"15min":16.004000000000001},"last":11.754},{"interface":"front","average":{"1min":1.339,"5min":1.1120000000000001,"15min":1.2470000000000001},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.32000000000000001},"max":{"1min":4.867,"5min":8.2040000000000006,"15min":8.2040000000000006},"last":11.766999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.044,"15min":0.89000000000000001},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.25900000000000001},"max":{"1min":20.827000000000002,"5min":20.827000000000002,"15min":20.827000000000002},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0169999999999999,"15min":0.82899999999999996},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":10.022,"5min":10.022,"15min":10.022},"last":1.0249999999999999}]},{"osd":5,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.8079999999999998,"5min":1.448,"15min":1.073},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":41.658000000000001,"5min":41.658000000000001,"15min":41.658000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":3.9060000000000001,"5min":1.423,"15min":1.0580000000000001},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.29199999999999998},"max":{"1min":41.634999999999998,"5min":41.634999999999998,"15min":41.634999999999998},"last":1.129}]},{"osd":7,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.012,"15min":0.93700000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":4.7960000000000003,"5min":5.2859999999999996,"15min":5.625},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.82499999999999996,"15min":0.874},"min":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.318},"max":{"1min":3.835,"5min":4.5359999999999996,"15min":7.0810000000000004},"last":0.45100000000000001}]}]},{"osd":5,"up_from":16,"seq":68719482916,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7277404,"kb_used_data":5634448,"kb_used_omap":463,"kb_used_meta":1642480,"kb_avail":87094436,"statfs":{"total":96636764160,"available":89184702464,"internally_reserved":0,"allocated":5769674752,"data_stored":5756769543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475086,"internal_metadata":1681899570},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.66300000000000003,"15min":0.73799999999999999},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.23200000000000001},"max":{"1min":5.2619999999999996,"5min":5.2619999999999996,"15min":6.2859999999999996},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.90700000000000003,"15min":0.84299999999999997},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":3.6600000000000001,"5min":5.5810000000000004,"15min":6.4829999999999997},"last":0.67700000000000005}]},{"osd":1,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.0549999999999999,"15min":1.018},"min":{"1min":0.27700000000000002,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":5.8109999999999999,"5min":9.2159999999999993,"15min":9.2159999999999993},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.92500000000000004,"15min":0.875},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":8.6959999999999997},"last":0.76700000000000002}]},{"osd":2,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.92700000000000005,"15min":0.90200000000000002},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28699999999999998},"max":{"1min":5.0919999999999996,"5min":5.4290000000000003,"15min":6.3460000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.196,"15min":1.145},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":5.8810000000000002,"5min":5.8810000000000002,"15min":6.3540000000000001},"last":0.90700000000000003}]},{"osd":3,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":1.181,"15min":1.1160000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.307},"max":{"1min":20.481000000000002,"5min":20.481000000000002,"15min":20.481000000000002},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.25,"15min":1.1870000000000001},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.29099999999999998},"max":{"1min":5.8520000000000003,"5min":5.8520000000000003,"15min":6.444},"last":1.024}]},{"osd":4,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.66900000000000004,"15min":0.68899999999999995},"min":{"1min":0.314,"5min":0.249,"15min":0.249},"max":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":6.2149999999999999},"last":1.125},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79700000000000004,"15min":0.78400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.3490000000000002,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.2749999999999999}]},{"osd":6,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":1.1319999999999999,"15min":0.98699999999999999},"min":{"1min":0.38700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":27.823,"5min":27.823,"15min":27.823},"last":1.0960000000000001},{"interface":"front","average":{"1min":2.5529999999999999,"5min":1.0109999999999999,"15min":0.86199999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":33.023000000000003,"5min":33.023000000000003,"15min":33.023000000000003},"last":1.3280000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.5230000000000001,"5min":1.4139999999999999,"15min":1.0029999999999999},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":42.939,"5min":42.939,"15min":42.939},"last":1.3580000000000001},{"interface":"front","average":{"1min":2.234,"5min":1.0629999999999999,"15min":1.0209999999999999},"min":{"1min":0.44600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":18.196999999999999,"5min":18.196999999999999,"15min":18.196999999999999},"last":1.222}]}]},{"osd":4,"up_from":16,"seq":68719482914,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8047568,"kb_used_data":6360644,"kb_used_omap":362,"kb_used_meta":1686549,"kb_avail":86324272,"statfs":{"total":96636764160,"available":88396054528,"internally_reserved":0,"allocated":6513299456,"data_stored":6498755697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":371446,"internal_metadata":1727026442},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.57599999999999996,"15min":0.69199999999999995},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.246},"max":{"1min":1.2909999999999999,"5min":4.8220000000000001,"15min":6.8559999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.244,"5min":0.94799999999999995,"15min":0.91400000000000003},"min":{"1min":0.309,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.5,"5min":7.1079999999999997,"15min":8.9420000000000002},"last":0.81799999999999995}]},{"osd":1,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.0249999999999999,"15min":1.0489999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.251},"max":{"1min":6.5339999999999998,"5min":7.3399999999999999,"15min":7.3399999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81100000000000005,"15min":1.161},"min":{"1min":0.28699999999999998,"5min":0.27300000000000002,"15min":0.25900000000000001},"max":{"1min":3.6709999999999998,"5min":7.1849999999999996,"15min":27.398},"last":0.35899999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.95899999999999996,"15min":0.98499999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":5.2569999999999997,"5min":7.9189999999999996,"15min":7.9189999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.964,"5min":1.5229999999999999,"15min":1.4219999999999999},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.29299999999999998},"max":{"1min":6.641,"5min":10.103,"15min":10.103},"last":0.74399999999999999}]},{"osd":3,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.234,"15min":1.333},"min":{"1min":0.33700000000000002,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":5.1349999999999998,"5min":11.417999999999999,"15min":12.627000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.0629999999999999,"15min":1.2110000000000001},"min":{"1min":0.32400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":4.2089999999999996,"5min":8.1259999999999994,"15min":10.43},"last":0.39400000000000002}]},{"osd":5,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81599999999999995,"15min":0.78000000000000003},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.4660000000000002,"5min":6.4660000000000002,"15min":8.7249999999999996},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.61199999999999999,"15min":0.73899999999999999},"min":{"1min":0.33700000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.105,"5min":4.2629999999999999,"15min":6.6559999999999997},"last":0.53700000000000003}]},{"osd":6,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.72199999999999998,"15min":0.80100000000000005},"min":{"1min":0.34699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.1990000000000001,"5min":7.5999999999999996,"15min":7.5999999999999996},"last":0.44},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.004,"15min":0.91000000000000003},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.34000000000000002},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"last":0.87}]},{"osd":7,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.74199999999999999,"15min":0.84499999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":6.1219999999999999,"5min":6.1219999999999999,"15min":6.6879999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.88400000000000001,"15min":0.91300000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80000000000000004,"5min":5.2919999999999998,"15min":6.9379999999999997},"last":1.246}]}]},{"osd":3,"up_from":15,"seq":64424515608,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8242848,"kb_used_data":6540564,"kb_used_omap":132,"kb_used_meta":1702139,"kb_avail":86128992,"statfs":{"total":96636764160,"available":88196087808,"internally_reserved":0,"allocated":6697537536,"data_stored":6682960825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136113,"internal_metadata":1742990415},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":1.1759999999999999,"15min":1.1319999999999999},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.20000000000000001},"max":{"1min":21.396999999999998,"5min":21.396999999999998,"15min":21.396999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.393,"5min":0.93799999999999994,"15min":0.92000000000000004},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":12.132,"5min":12.132,"15min":12.132},"last":0.749}]},{"osd":1,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":1.339,"15min":1.1479999999999999},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.215},"max":{"1min":29.577000000000002,"5min":29.577000000000002,"15min":29.577000000000002},"last":0.879},{"interface":"front","average":{"1min":2.0870000000000002,"5min":1.4950000000000001,"15min":1.373},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":12.877000000000001,"5min":12.877000000000001,"15min":12.877000000000001},"last":0.70199999999999996}]},{"osd":2,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":1.3200000000000001,"15min":1.395},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.253},"max":{"1min":12.920999999999999,"5min":12.920999999999999,"15min":12.920999999999999},"last":1.109},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.3380000000000001,"15min":1.177},"min":{"1min":0.34999999999999998,"5min":0.29799999999999999,"15min":0.27500000000000002},"max":{"1min":17.669,"5min":17.669,"15min":17.669},"last":1.1559999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.6339999999999999,"5min":1.488,"15min":1.2769999999999999},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":13.095000000000001,"5min":13.228,"15min":13.228},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.161,"15min":1.0860000000000001},"min":{"1min":0.35899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":12.166,"5min":12.166,"15min":12.166},"last":1.198}]},{"osd":5,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.3889999999999998,"5min":1.4710000000000001,"15min":1.234},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":29.622,"5min":29.622,"15min":29.622},"last":1.044},{"interface":"front","average":{"1min":3.5619999999999998,"5min":1.6699999999999999,"15min":1.409},"min":{"1min":0.38100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":21.038,"5min":21.038,"15min":21.038},"last":1.244}]},{"osd":6,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.1539999999999999,"15min":1.105},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.34999999999999998},"max":{"1min":4.0380000000000003,"5min":5.5220000000000002,"15min":10.683},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.786,"5min":1.296,"15min":1.1779999999999999},"min":{"1min":0.47299999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":15.619999999999999,"5min":15.619999999999999,"15min":15.619999999999999},"last":1.286}]},{"osd":7,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.2989999999999999,"5min":1.71,"15min":1.4810000000000001},"min":{"1min":0.42099999999999999,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":18.16,"5min":18.16,"15min":18.16},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.643,"15min":1.288},"min":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.41799999999999998},"max":{"1min":10.282999999999999,"5min":22.334,"15min":22.334},"last":1.613}]}]},{"osd":1,"up_from":15,"seq":64424515613,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2659224,"kb_used_data":825552,"kb_used_omap":997,"kb_used_meta":1832666,"kb_avail":91712616,"statfs":{"total":96636764160,"available":93913718784,"internally_reserved":0,"allocated":845365248,"data_stored":843373484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021567,"internal_metadata":1876650369},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":130,"apply_latency_ms":130,"commit_latency_ns":130000000,"apply_latency_ns":130000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.81499999999999995,"15min":0.89000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.5490000000000004,"5min":9.8010000000000002,"15min":9.8010000000000002},"last":0.377},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.089,"15min":1.196},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.214},"max":{"1min":30.109999999999999,"5min":30.109999999999999,"15min":30.109999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5129999999999999,"5min":1.2869999999999999,"15min":1.202},"min":{"1min":0.30499999999999999,"5min":0.23300000000000001,"15min":0.224},"max":{"1min":25.259,"5min":25.259,"15min":25.259},"last":0.876},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.873,"15min":0.998},"min":{"1min":0.29199999999999998,"5min":0.245,"15min":0.22500000000000001},"max":{"1min":5.4580000000000002,"5min":9.7720000000000002,"15min":9.7720000000000002},"last":0.57699999999999996}]},{"osd":3,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.8199999999999998,"5min":1.218,"15min":1.2929999999999999},"min":{"1min":0.314,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":30.460999999999999,"5min":30.460999999999999,"15min":30.460999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.8120000000000001,"5min":1.4039999999999999,"15min":1.3080000000000001},"min":{"1min":0.311,"5min":0.28999999999999998,"15min":0.25},"max":{"1min":6.6210000000000004,"5min":8.0980000000000008,"15min":9.6460000000000008},"last":0.81899999999999995}]},{"osd":4,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1519999999999999,"15min":0.96899999999999997},"min":{"1min":0.34999999999999998,"5min":0.31,"15min":0.27300000000000002},"max":{"1min":5.5190000000000001,"5min":9.6669999999999998,"15min":10.122},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.0780000000000001,"15min":1.0600000000000001},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":0.749}]},{"osd":5,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.127,"15min":0.99199999999999999},"min":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.315},"max":{"1min":5.492,"5min":9.7110000000000003,"15min":9.7110000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":2.851,"5min":1.218,"15min":1.2490000000000001},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":30.277999999999999,"5min":30.277999999999999,"15min":30.277999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1830000000000001,"15min":1.1819999999999999},"min":{"1min":0.46700000000000003,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":4.8200000000000003,"5min":6.3979999999999997,"15min":8.8840000000000003},"last":0.78800000000000003},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.5109999999999999,"15min":1.1779999999999999},"min":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38700000000000001},"max":{"1min":19.016999999999999,"5min":19.016999999999999,"15min":19.016999999999999},"last":0.63400000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.9620000000000002,"5min":1.401,"15min":1.284},"min":{"1min":0.52600000000000002,"5min":0.40699999999999997,"15min":0.378},"max":{"1min":30.242999999999999,"5min":30.242999999999999,"15min":30.242999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.659,"5min":1.2869999999999999,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":4.9240000000000004,"5min":6.46,"15min":8.9589999999999996},"last":0.84799999999999998}]}]},{"osd":0,"up_from":15,"seq":64424515615,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5625040,"kb_used_data":4244484,"kb_used_omap":422,"kb_used_meta":1380121,"kb_avail":88746800,"statfs":{"total":96636764160,"available":90876723200,"internally_reserved":0,"allocated":4346351616,"data_stored":4336938095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":432402,"internal_metadata":1413244654},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":0.88400000000000001,"15min":0.97199999999999998},"min":{"1min":0.20200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":8.3789999999999996,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.96199999999999997,"15min":1.181},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1079999999999997,"5min":5.1470000000000002,"15min":8.1039999999999992},"last":0.247}]},{"osd":2,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.159,"15min":1.194},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.2759999999999998,"5min":7.2759999999999998,"15min":10.146000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.2130000000000001,"15min":1.2889999999999999},"min":{"1min":0.26600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.4770000000000003,"5min":13.676,"15min":13.676},"last":0.39200000000000002}]},{"osd":3,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.2929999999999999,"15min":1.2749999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.25},"max":{"1min":2.5680000000000001,"5min":13.721,"15min":13.721},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.0840000000000001,"15min":1.1870000000000001},"min":{"1min":0.37,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":6.2089999999999996,"5min":8.0570000000000004,"15min":10.037000000000001},"last":1.5880000000000001}]},{"osd":4,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.145,"15min":1.1919999999999999},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.0579999999999998,"5min":13.755000000000001,"15min":13.755000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.0389999999999999,"15min":0.93200000000000005},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.29199999999999998},"max":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":7.3540000000000001},"last":0.30299999999999999}]},{"osd":5,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":0.95499999999999996,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.32700000000000001},"max":{"1min":6.5090000000000003,"5min":6.5090000000000003,"15min":7.4349999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.0089999999999999,"15min":1.149},"min":{"1min":0.379,"5min":0.31,"15min":0.31},"max":{"1min":4.1589999999999998,"5min":5.101,"15min":8.1820000000000004},"last":0.58999999999999997}]},{"osd":6,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.25,"15min":1.3069999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.34300000000000003},"max":{"1min":6.202,"5min":13.819000000000001,"15min":13.819000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.1499999999999999,"15min":1.087},"min":{"1min":0.38,"5min":0.38,"15min":0.378},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":0.46300000000000002}]},{"osd":7,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.087,"15min":1.2030000000000001},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":4.2119999999999997,"5min":5.1760000000000002,"15min":8.2110000000000003},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.266,"15min":1.4059999999999999},"min":{"1min":0.39500000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.242,"5min":13.786,"15min":13.786},"last":0.44}]}]},{"osd":2,"up_from":14,"seq":60129548316,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5800052,"kb_used_data":4523496,"kb_used_omap":313,"kb_used_meta":1276230,"kb_avail":88571788,"statfs":{"total":96636764160,"available":90697510912,"internally_reserved":0,"allocated":4632059904,"data_stored":4622129985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":320629,"internal_metadata":1306860427},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":0.96099999999999997,"15min":1.0049999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.8399999999999999,"5min":7.5919999999999996,"15min":8.1500000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3120000000000001,"15min":1.1439999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":10.301,"5min":10.301,"15min":10.301},"last":0.59499999999999997}]},{"osd":1,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0329999999999999,"15min":1.103},"min":{"1min":0.26600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":4.9039999999999999,"5min":6.8010000000000002,"15min":7.7320000000000002},"last":0.438},{"interface":"front","average":{"1min":1.181,"5min":1.333,"15min":1.27},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":10.345000000000001,"5min":10.345000000000001,"15min":10.345000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":1.446,"15min":1.4199999999999999},"min":{"1min":0.26700000000000002,"5min":0.24299999999999999,"15min":0.23999999999999999},"max":{"1min":10.706,"5min":10.706,"15min":10.706},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.042,"15min":1.1990000000000001},"min":{"1min":0.36699999999999999,"5min":0.29699999999999999,"15min":0.26100000000000001},"max":{"1min":4.9340000000000002,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.58899999999999997}]},{"osd":4,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.999,"15min":1.0920000000000001},"min":{"1min":0.374,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":4.7990000000000004,"5min":6.1870000000000003,"15min":8.0860000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.29,"15min":1.159},"min":{"1min":0.375,"5min":0.314,"15min":0.314},"max":{"1min":4.1630000000000003,"5min":9.4399999999999995,"15min":9.4399999999999995},"last":0.66000000000000003}]},{"osd":5,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.383,"15min":1.2649999999999999},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.9779999999999998,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.341,"15min":1.3759999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":7.8929999999999998,"5min":7.8929999999999998,"15min":10.366},"last":0.40899999999999997}]},{"osd":6,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":1.427,"15min":1.431},"min":{"1min":0.45900000000000002,"5min":0.39600000000000002,"15min":0.376},"max":{"1min":7.9770000000000003,"5min":7.9770000000000003,"15min":10.412000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.075,"15min":1.196},"min":{"1min":0.50800000000000001,"5min":0.36499999999999999,"15min":0.35999999999999999},"max":{"1min":4.8719999999999999,"5min":7.8959999999999999,"15min":8.0350000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.444,"15min":1.2909999999999999},"min":{"1min":0.44400000000000001,"5min":0.40100000000000002,"15min":0.37},"max":{"1min":5.8949999999999996,"5min":9.5489999999999995,"15min":9.5489999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":2.4449999999999998,"5min":1.456,"15min":1.4670000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":7.9429999999999996,"5min":7.9429999999999996,"15min":10.446},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-17T06:32:19.037 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-17T06:32:19.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-17T06:32:19.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b51036640 1 Processor -- start 2023-12-17T06:32:19.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b51036640 1 -- start start 2023-12-17T06:32:19.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b51036640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6b4c0723e0 0x7f6b4c0ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:19.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b51036640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c15b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:19.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b51036640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4c15bdf0 con 0x7f6b4c15ec10 2023-12-17T06:32:19.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b51036640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4c070f10 con 0x7f6b4c0723e0 2023-12-17T06:32:19.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b51036640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0711f0 con 0x7f6b4c15b440 2023-12-17T06:32:19.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b4ad9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f6b4c15ec10 0x7f6b4c071cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:48622/0 (socket says 172.21.15.28:48622) 2023-12-17T06:32:19.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b4a59c640 1 --2- 172.21.15.28:0/1074525740 >> v2:172.21.15.28:3300/0 conn(0x7f6b4c0723e0 0x7f6b4c0ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:19.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b4ad9d640 1 -- 172.21.15.28:0/1074525740 learned_addr learned my addr 172.21.15.28:0/1074525740 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:19.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b4b59e640 1 --2- 172.21.15.28:0/1074525740 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c15b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:19.328 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b4b59e640 1 -- 172.21.15.28:0/1074525740 >> v2:172.21.15.28:3300/0 conn(0x7f6b4c0723e0 msgr2=0x7f6b4c0ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:19.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b4b59e640 1 --2- 172.21.15.28:0/1074525740 >> v2:172.21.15.28:3300/0 conn(0x7f6b4c0723e0 0x7f6b4c0ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b4b59e640 1 -- 172.21.15.28:0/1074525740 >> v1:172.21.15.122:6789/0 conn(0x7f6b4c15ec10 legacy=0x7f6b4c071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:19.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.318+0000 7f6b4b59e640 1 -- 172.21.15.28:0/1074525740 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c0714d0 con 0x7f6b4c15b440 2023-12-17T06:32:19.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.322+0000 7f6b4b59e640 1 --2- 172.21.15.28:0/1074525740 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c15b8b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6b400097c0 tx=0x7f6b4001c280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b08c47c4d15e577 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:19.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b49d9b640 1 -- 172.21.15.28:0/1074525740 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b400ab600 con 0x7f6b4c15b440 2023-12-17T06:32:19.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b49d9b640 1 -- 172.21.15.28:0/1074525740 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b400a7050 con 0x7f6b4c15b440 2023-12-17T06:32:19.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b49d9b640 1 -- 172.21.15.28:0/1074525740 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b400ab980 con 0x7f6b4c15b440 2023-12-17T06:32:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b51036640 1 -- 172.21.15.28:0/1074525740 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 msgr2=0x7f6b4c15b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b51036640 1 --2- 172.21.15.28:0/1074525740 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c15b8b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6b400097c0 tx=0x7f6b4001c280 comp rx=0 tx=0).stop 2023-12-17T06:32:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b51036640 1 -- 172.21.15.28:0/1074525740 shutdown_connections 2023-12-17T06:32:19.335 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b51036640 1 --2- 172.21.15.28:0/1074525740 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c15b8b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.335 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b51036640 1 --2- 172.21.15.28:0/1074525740 >> v2:172.21.15.28:3300/0 conn(0x7f6b4c0723e0 0x7f6b4c0ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.335 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b51036640 1 -- 172.21.15.28:0/1074525740 >> 172.21.15.28:0/1074525740 conn(0x7f6b4c06c2c0 msgr2=0x7f6b4c06c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:19.335 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b51036640 1 -- 172.21.15.28:0/1074525740 shutdown_connections 2023-12-17T06:32:19.335 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.326+0000 7f6b51036640 1 -- 172.21.15.28:0/1074525740 wait complete. 2023-12-17T06:32:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b51036640 1 Processor -- start 2023-12-17T06:32:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b51036640 1 -- start start 2023-12-17T06:32:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b51036640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c0723e0 0x7f6b4c1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b51036640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c0723e0 0x7f6b4c1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c0723e0 0x7f6b4c1499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36652/0 (socket says 172.21.15.28:36652) 2023-12-17T06:32:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4a59c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4a59c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37236/0 (socket says 172.21.15.28:37236) 2023-12-17T06:32:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4ad9d640 1 -- 172.21.15.28:0/3581783605 learned_addr learned my addr 172.21.15.28:0/3581783605 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b51036640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c15ec10 0x7f6b4c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c149ee0 con 0x7f6b4c0723e0 2023-12-17T06:32:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c1576c0 con 0x7f6b4c15ec10 2023-12-17T06:32:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0b2160 con 0x7f6b4c15b440 2023-12-17T06:32:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4b59e640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c15ec10 0x7f6b4c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4ad9d640 1 -- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 msgr2=0x7f6b4c156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4ad9d640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4ad9d640 1 -- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c15ec10 msgr2=0x7f6b4c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4ad9d640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c15ec10 0x7f6b4c0b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4ad9d640 1 -- 172.21.15.28:0/3581783605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b40009420 con 0x7f6b4c0723e0 2023-12-17T06:32:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.330+0000 7f6b4a59c640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-17T06:32:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.334+0000 7f6b4ad9d640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c0723e0 0x7f6b4c1499a0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400caf0 tx=0x7f6b44002100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:19.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.346+0000 7f6b337fe640 1 -- 172.21.15.28:0/3581783605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400a760 con 0x7f6b4c0723e0 2023-12-17T06:32:19.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.346+0000 7f6b337fe640 1 -- 172.21.15.28:0/3581783605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4400a8c0 con 0x7f6b4c0723e0 2023-12-17T06:32:19.351 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.346+0000 7f6b337fe640 1 -- 172.21.15.28:0/3581783605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400ac70 con 0x7f6b4c0723e0 2023-12-17T06:32:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.350+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4c0b2440 con 0x7f6b4c0723e0 2023-12-17T06:32:19.355 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.350+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c0b2a00 con 0x7f6b4c0723e0 2023-12-17T06:32:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.350+0000 7f6b337fe640 1 -- 172.21.15.28:0/3581783605 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6b440070d0 con 0x7f6b4c0723e0 2023-12-17T06:32:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.350+0000 7f6b337fe640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b380664c0 0x7f6b38068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:19.358 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.350+0000 7f6b337fe640 1 -- 172.21.15.28:0/3581783605 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6b44083290 con 0x7f6b4c0723e0 2023-12-17T06:32:19.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.350+0000 7f6b4a59c640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b380664c0 0x7f6b38068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:19.359 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.354+0000 7f6b4a59c640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b380664c0 0x7f6b38068980 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c165e10 tx=0x7f6b3c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:19.371 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.366+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4c0b2bf0 con 0x7f6b4c0723e0 2023-12-17T06:32:19.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.374+0000 7f6b337fe640 1 -- 172.21.15.28:0/3581783605 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f6b44018080 con 0x7f6b4c0723e0 2023-12-17T06:32:19.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.730+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b4c167f50 con 0x7f6b380664c0 2023-12-17T06:32:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.734+0000 7f6b337fe640 1 -- 172.21.15.28:0/3581783605 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48990 (secure 0 0 0) 0x7f6b3c0119d0 con 0x7f6b380664c0 2023-12-17T06:32:19.742 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-17T06:32:19.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.742+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b380664c0 msgr2=0x7f6b38068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:19.748 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.742+0000 7f6b51036640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b380664c0 0x7f6b38068980 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c165e10 tx=0x7f6b3c008040 comp rx=0 tx=0).stop 2023-12-17T06:32:19.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.742+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c0723e0 msgr2=0x7f6b4c1499a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:19.749 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.742+0000 7f6b51036640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c0723e0 0x7f6b4c1499a0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400caf0 tx=0x7f6b44002100 comp rx=0 tx=0).stop 2023-12-17T06:32:19.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.746+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 shutdown_connections 2023-12-17T06:32:19.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.746+0000 7f6b51036640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f6b380664c0 0x7f6b38068980 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.753 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.746+0000 7f6b51036640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b4c15ec10 0x7f6b4c0b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.746+0000 7f6b51036640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f6b4c15b440 0x7f6b4c156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.746+0000 7f6b51036640 1 --2- 172.21.15.28:0/3581783605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c0723e0 0x7f6b4c1499a0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:19.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.746+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 >> 172.21.15.28:0/3581783605 conn(0x7f6b4c06c2c0 msgr2=0x7f6b4c0afaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:19.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.746+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 shutdown_connections 2023-12-17T06:32:19.754 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:19.746+0000 7f6b51036640 1 -- 172.21.15.28:0/3581783605 wait complete. 2023-12-17T06:32:19.775 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":15471,"stamp":"2023-12-17T06:32:18.771238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":46931448,"kb_used_data":34142368,"kb_used_omap":3694,"kb_used_meta":12785297,"kb_avail":708043272,"statfs":{"total":773094113280,"available":725036310528,"internally_reserved":0,"allocated":34961784832,"data_stored":34884853364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3783506,"internal_metadata":13092144302},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":250,"apply_latency_ms":250,"commit_latency_ns":250000000,"apply_latency_ns":250000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-43516366526,"num_objects":-35384,"num_object_clones":0,"num_object_copies":-70768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-35384,"num_whiteouts":0,"num_read":-1890626,"num_read_kb":-132487568,"num_write":-7495234,"num_write_kb":-162617007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2360,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001630"},"pg_stats":[{"pgid":"1.0","version":"36'161","reported_seq":484,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-12-17T06:32:09.754483+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-17T06:32:09.754483+0000","last_peered":"2023-12-17T06:32:09.754483+0000","last_clean":"2023-12-17T06:32:09.754483+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-17T06:32:09.754483+0000","last_undegraded":"2023-12-17T06:32:09.754483+0000","last_fullsized":"2023-12-17T06:32:09.754483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719482916,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7340040,"kb_used_data":5844604,"kb_used_omap":20,"kb_used_meta":1495403,"kb_avail":87031800,"statfs":{"total":96636764160,"available":89120563200,"internally_reserved":0,"allocated":5984874496,"data_stored":5972031162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20951,"internal_metadata":1531293225},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.9750000000000001,"5min":1.1080000000000001,"15min":0.88500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":15.872,"5min":15.872,"15min":15.872},"last":2.4199999999999999},{"interface":"front","average":{"1min":2.355,"5min":1.238,"15min":0.96999999999999997},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":22.158000000000001,"5min":22.158000000000001,"15min":22.158000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.97099999999999997,"15min":0.82199999999999995},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":8.2279999999999998,"5min":8.2279999999999998,"15min":11.856},"last":2.4689999999999999},{"interface":"front","average":{"1min":2.6429999999999998,"5min":1.4390000000000001,"15min":1.1599999999999999},"min":{"1min":0.30199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":19.102,"5min":19.102,"15min":19.102},"last":0.49399999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.327,"15min":1.196},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":19.332000000000001,"5min":19.332000000000001,"15min":19.332000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.593,"5min":1.518,"15min":1.2969999999999999},"min":{"1min":0.33600000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":15.898,"5min":15.898,"15min":15.898},"last":2.3370000000000002}]},{"osd":3,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":3.137,"5min":1.635,"15min":1.365},"min":{"1min":0.35399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":24.094999999999999,"5min":24.094999999999999,"15min":24.094999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.3330000000000002,"5min":1.5529999999999999,"15min":1.26},"min":{"1min":0.39300000000000002,"5min":0.32500000000000001,"15min":0.28999999999999998},"max":{"1min":15.837,"5min":15.837,"15min":15.837},"last":2.3730000000000002}]},{"osd":4,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.5089999999999999,"5min":1.141,"15min":0.86399999999999999},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27100000000000002},"max":{"1min":16.614000000000001,"5min":16.614000000000001,"15min":16.614000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.2989999999999999,"5min":0.83599999999999997,"15min":0.78600000000000003},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":8.2699999999999996,"5min":8.2699999999999996,"15min":8.2699999999999996},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":0.93100000000000005,"15min":0.88900000000000001},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":8.1969999999999992,"5min":8.1969999999999992,"15min":11.741},"last":2.2440000000000002},{"interface":"front","average":{"1min":2.8210000000000002,"5min":1.2929999999999999,"15min":1.024},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":27.901,"5min":27.901,"15min":27.901},"last":0.83299999999999996}]},{"osd":6,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.0860000000000001,"15min":0.90200000000000002},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":0.995},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.0660000000000001,"15min":0.95099999999999996},"min":{"1min":0.36299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.946,"5min":11.946,"15min":11.946},"last":0.56200000000000006}]}]},{"osd":6,"up_from":16,"seq":68719482916,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1939272,"kb_used_data":168576,"kb_used_omap":981,"kb_used_meta":1769706,"kb_avail":92432568,"statfs":{"total":96636764160,"available":94650949632,"internally_reserved":0,"allocated":172621824,"data_stored":171894573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1005312,"internal_metadata":1812179200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":96,"apply_latency_ms":96,"commit_latency_ns":96000000,"apply_latency_ns":96000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2879999999999998,"5min":1.2549999999999999,"15min":0.88900000000000001},"min":{"1min":0.26600000000000001,"5min":0.24399999999999999,"15min":0.22500000000000001},"max":{"1min":38.720999999999997,"5min":38.720999999999997,"15min":38.720999999999997},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.0669999999999999,"15min":0.90200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26500000000000001,"15min":0.25},"max":{"1min":5.3920000000000003,"5min":8.0779999999999994,"15min":8.0779999999999994},"last":0.41299999999999998}]},{"osd":1,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.1850000000000001,"15min":1.1990000000000001},"min":{"1min":0.26800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.2619999999999996,"5min":5.077,"15min":8.9410000000000007},"last":1.083},{"interface":"front","average":{"1min":1.639,"5min":1.1419999999999999,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":4.7590000000000003,"5min":8.798,"15min":8.798},"last":0.998}]},{"osd":2,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.3819999999999999,"15min":1.167},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.5819999999999999,"5min":8.9090000000000007,"15min":8.9090000000000007},"last":11.710000000000001},{"interface":"front","average":{"1min":1.496,"5min":1.425,"15min":1.236},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.29599999999999999},"max":{"1min":4.9790000000000001,"5min":8.8179999999999996,"15min":10.401999999999999},"last":11.788}]},{"osd":3,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2029999999999998,"5min":1.8009999999999999,"15min":1.369},"min":{"1min":0.38700000000000001,"5min":0.31,"15min":0.31},"max":{"1min":16.004000000000001,"5min":16.004000000000001,"15min":16.004000000000001},"last":11.754},{"interface":"front","average":{"1min":1.339,"5min":1.1120000000000001,"15min":1.2470000000000001},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.32000000000000001},"max":{"1min":4.867,"5min":8.2040000000000006,"15min":8.2040000000000006},"last":11.766999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.044,"15min":0.89000000000000001},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.25900000000000001},"max":{"1min":20.827000000000002,"5min":20.827000000000002,"15min":20.827000000000002},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0169999999999999,"15min":0.82899999999999996},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":10.022,"5min":10.022,"15min":10.022},"last":1.0249999999999999}]},{"osd":5,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.8079999999999998,"5min":1.448,"15min":1.073},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":41.658000000000001,"5min":41.658000000000001,"15min":41.658000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":3.9060000000000001,"5min":1.423,"15min":1.0580000000000001},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.29199999999999998},"max":{"1min":41.634999999999998,"5min":41.634999999999998,"15min":41.634999999999998},"last":1.129}]},{"osd":7,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.012,"15min":0.93700000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":4.7960000000000003,"5min":5.2859999999999996,"15min":5.625},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.82499999999999996,"15min":0.874},"min":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.318},"max":{"1min":3.835,"5min":4.5359999999999996,"15min":7.0810000000000004},"last":0.45100000000000001}]}]},{"osd":5,"up_from":16,"seq":68719482916,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7277404,"kb_used_data":5634448,"kb_used_omap":463,"kb_used_meta":1642480,"kb_avail":87094436,"statfs":{"total":96636764160,"available":89184702464,"internally_reserved":0,"allocated":5769674752,"data_stored":5756769543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475086,"internal_metadata":1681899570},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.66300000000000003,"15min":0.73799999999999999},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.23200000000000001},"max":{"1min":5.2619999999999996,"5min":5.2619999999999996,"15min":6.2859999999999996},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.90700000000000003,"15min":0.84299999999999997},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":3.6600000000000001,"5min":5.5810000000000004,"15min":6.4829999999999997},"last":0.67700000000000005}]},{"osd":1,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.0549999999999999,"15min":1.018},"min":{"1min":0.27700000000000002,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":5.8109999999999999,"5min":9.2159999999999993,"15min":9.2159999999999993},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.92500000000000004,"15min":0.875},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":8.6959999999999997},"last":0.76700000000000002}]},{"osd":2,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.92700000000000005,"15min":0.90200000000000002},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28699999999999998},"max":{"1min":5.0919999999999996,"5min":5.4290000000000003,"15min":6.3460000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.196,"15min":1.145},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":5.8810000000000002,"5min":5.8810000000000002,"15min":6.3540000000000001},"last":0.90700000000000003}]},{"osd":3,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":1.181,"15min":1.1160000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.307},"max":{"1min":20.481000000000002,"5min":20.481000000000002,"15min":20.481000000000002},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.25,"15min":1.1870000000000001},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.29099999999999998},"max":{"1min":5.8520000000000003,"5min":5.8520000000000003,"15min":6.444},"last":1.024}]},{"osd":4,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.66900000000000004,"15min":0.68899999999999995},"min":{"1min":0.314,"5min":0.249,"15min":0.249},"max":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":6.2149999999999999},"last":1.125},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79700000000000004,"15min":0.78400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.3490000000000002,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.2749999999999999}]},{"osd":6,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":1.1319999999999999,"15min":0.98699999999999999},"min":{"1min":0.38700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":27.823,"5min":27.823,"15min":27.823},"last":1.0960000000000001},{"interface":"front","average":{"1min":2.5529999999999999,"5min":1.0109999999999999,"15min":0.86199999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":33.023000000000003,"5min":33.023000000000003,"15min":33.023000000000003},"last":1.3280000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.5230000000000001,"5min":1.4139999999999999,"15min":1.0029999999999999},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":42.939,"5min":42.939,"15min":42.939},"last":1.3580000000000001},{"interface":"front","average":{"1min":2.234,"5min":1.0629999999999999,"15min":1.0209999999999999},"min":{"1min":0.44600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":18.196999999999999,"5min":18.196999999999999,"15min":18.196999999999999},"last":1.222}]}]},{"osd":4,"up_from":16,"seq":68719482914,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8047568,"kb_used_data":6360644,"kb_used_omap":362,"kb_used_meta":1686549,"kb_avail":86324272,"statfs":{"total":96636764160,"available":88396054528,"internally_reserved":0,"allocated":6513299456,"data_stored":6498755697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":371446,"internal_metadata":1727026442},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.57599999999999996,"15min":0.69199999999999995},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.246},"max":{"1min":1.2909999999999999,"5min":4.8220000000000001,"15min":6.8559999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.244,"5min":0.94799999999999995,"15min":0.91400000000000003},"min":{"1min":0.309,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.5,"5min":7.1079999999999997,"15min":8.9420000000000002},"last":0.81799999999999995}]},{"osd":1,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.0249999999999999,"15min":1.0489999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.251},"max":{"1min":6.5339999999999998,"5min":7.3399999999999999,"15min":7.3399999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81100000000000005,"15min":1.161},"min":{"1min":0.28699999999999998,"5min":0.27300000000000002,"15min":0.25900000000000001},"max":{"1min":3.6709999999999998,"5min":7.1849999999999996,"15min":27.398},"last":0.35899999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.95899999999999996,"15min":0.98499999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":5.2569999999999997,"5min":7.9189999999999996,"15min":7.9189999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.964,"5min":1.5229999999999999,"15min":1.4219999999999999},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.29299999999999998},"max":{"1min":6.641,"5min":10.103,"15min":10.103},"last":0.74399999999999999}]},{"osd":3,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.234,"15min":1.333},"min":{"1min":0.33700000000000002,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":5.1349999999999998,"5min":11.417999999999999,"15min":12.627000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.0629999999999999,"15min":1.2110000000000001},"min":{"1min":0.32400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":4.2089999999999996,"5min":8.1259999999999994,"15min":10.43},"last":0.39400000000000002}]},{"osd":5,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81599999999999995,"15min":0.78000000000000003},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.4660000000000002,"5min":6.4660000000000002,"15min":8.7249999999999996},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.61199999999999999,"15min":0.73899999999999999},"min":{"1min":0.33700000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.105,"5min":4.2629999999999999,"15min":6.6559999999999997},"last":0.53700000000000003}]},{"osd":6,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.72199999999999998,"15min":0.80100000000000005},"min":{"1min":0.34699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.1990000000000001,"5min":7.5999999999999996,"15min":7.5999999999999996},"last":0.44},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.004,"15min":0.91000000000000003},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.34000000000000002},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"last":0.87}]},{"osd":7,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.74199999999999999,"15min":0.84499999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":6.1219999999999999,"5min":6.1219999999999999,"15min":6.6879999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.88400000000000001,"15min":0.91300000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80000000000000004,"5min":5.2919999999999998,"15min":6.9379999999999997},"last":1.246}]}]},{"osd":3,"up_from":15,"seq":64424515608,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8242848,"kb_used_data":6540564,"kb_used_omap":132,"kb_used_meta":1702139,"kb_avail":86128992,"statfs":{"total":96636764160,"available":88196087808,"internally_reserved":0,"allocated":6697537536,"data_stored":6682960825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136113,"internal_metadata":1742990415},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":1.1759999999999999,"15min":1.1319999999999999},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.20000000000000001},"max":{"1min":21.396999999999998,"5min":21.396999999999998,"15min":21.396999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.393,"5min":0.93799999999999994,"15min":0.92000000000000004},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":12.132,"5min":12.132,"15min":12.132},"last":0.749}]},{"osd":1,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":1.339,"15min":1.1479999999999999},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.215},"max":{"1min":29.577000000000002,"5min":29.577000000000002,"15min":29.577000000000002},"last":0.879},{"interface":"front","average":{"1min":2.0870000000000002,"5min":1.4950000000000001,"15min":1.373},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":12.877000000000001,"5min":12.877000000000001,"15min":12.877000000000001},"last":0.70199999999999996}]},{"osd":2,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":1.3200000000000001,"15min":1.395},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.253},"max":{"1min":12.920999999999999,"5min":12.920999999999999,"15min":12.920999999999999},"last":1.109},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.3380000000000001,"15min":1.177},"min":{"1min":0.34999999999999998,"5min":0.29799999999999999,"15min":0.27500000000000002},"max":{"1min":17.669,"5min":17.669,"15min":17.669},"last":1.1559999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.6339999999999999,"5min":1.488,"15min":1.2769999999999999},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":13.095000000000001,"5min":13.228,"15min":13.228},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.161,"15min":1.0860000000000001},"min":{"1min":0.35899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":12.166,"5min":12.166,"15min":12.166},"last":1.198}]},{"osd":5,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.3889999999999998,"5min":1.4710000000000001,"15min":1.234},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":29.622,"5min":29.622,"15min":29.622},"last":1.044},{"interface":"front","average":{"1min":3.5619999999999998,"5min":1.6699999999999999,"15min":1.409},"min":{"1min":0.38100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":21.038,"5min":21.038,"15min":21.038},"last":1.244}]},{"osd":6,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.1539999999999999,"15min":1.105},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.34999999999999998},"max":{"1min":4.0380000000000003,"5min":5.5220000000000002,"15min":10.683},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.786,"5min":1.296,"15min":1.1779999999999999},"min":{"1min":0.47299999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":15.619999999999999,"5min":15.619999999999999,"15min":15.619999999999999},"last":1.286}]},{"osd":7,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.2989999999999999,"5min":1.71,"15min":1.4810000000000001},"min":{"1min":0.42099999999999999,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":18.16,"5min":18.16,"15min":18.16},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.643,"15min":1.288},"min":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.41799999999999998},"max":{"1min":10.282999999999999,"5min":22.334,"15min":22.334},"last":1.613}]}]},{"osd":1,"up_from":15,"seq":64424515613,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2659224,"kb_used_data":825552,"kb_used_omap":997,"kb_used_meta":1832666,"kb_avail":91712616,"statfs":{"total":96636764160,"available":93913718784,"internally_reserved":0,"allocated":845365248,"data_stored":843373484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021567,"internal_metadata":1876650369},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":130,"apply_latency_ms":130,"commit_latency_ns":130000000,"apply_latency_ns":130000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.81499999999999995,"15min":0.89000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.5490000000000004,"5min":9.8010000000000002,"15min":9.8010000000000002},"last":0.377},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.089,"15min":1.196},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.214},"max":{"1min":30.109999999999999,"5min":30.109999999999999,"15min":30.109999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5129999999999999,"5min":1.2869999999999999,"15min":1.202},"min":{"1min":0.30499999999999999,"5min":0.23300000000000001,"15min":0.224},"max":{"1min":25.259,"5min":25.259,"15min":25.259},"last":0.876},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.873,"15min":0.998},"min":{"1min":0.29199999999999998,"5min":0.245,"15min":0.22500000000000001},"max":{"1min":5.4580000000000002,"5min":9.7720000000000002,"15min":9.7720000000000002},"last":0.57699999999999996}]},{"osd":3,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.8199999999999998,"5min":1.218,"15min":1.2929999999999999},"min":{"1min":0.314,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":30.460999999999999,"5min":30.460999999999999,"15min":30.460999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.8120000000000001,"5min":1.4039999999999999,"15min":1.3080000000000001},"min":{"1min":0.311,"5min":0.28999999999999998,"15min":0.25},"max":{"1min":6.6210000000000004,"5min":8.0980000000000008,"15min":9.6460000000000008},"last":0.81899999999999995}]},{"osd":4,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1519999999999999,"15min":0.96899999999999997},"min":{"1min":0.34999999999999998,"5min":0.31,"15min":0.27300000000000002},"max":{"1min":5.5190000000000001,"5min":9.6669999999999998,"15min":10.122},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.0780000000000001,"15min":1.0600000000000001},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":0.749}]},{"osd":5,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.127,"15min":0.99199999999999999},"min":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.315},"max":{"1min":5.492,"5min":9.7110000000000003,"15min":9.7110000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":2.851,"5min":1.218,"15min":1.2490000000000001},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":30.277999999999999,"5min":30.277999999999999,"15min":30.277999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1830000000000001,"15min":1.1819999999999999},"min":{"1min":0.46700000000000003,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":4.8200000000000003,"5min":6.3979999999999997,"15min":8.8840000000000003},"last":0.78800000000000003},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.5109999999999999,"15min":1.1779999999999999},"min":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38700000000000001},"max":{"1min":19.016999999999999,"5min":19.016999999999999,"15min":19.016999999999999},"last":0.63400000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.9620000000000002,"5min":1.401,"15min":1.284},"min":{"1min":0.52600000000000002,"5min":0.40699999999999997,"15min":0.378},"max":{"1min":30.242999999999999,"5min":30.242999999999999,"15min":30.242999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.659,"5min":1.2869999999999999,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":4.9240000000000004,"5min":6.46,"15min":8.9589999999999996},"last":0.84799999999999998}]}]},{"osd":0,"up_from":15,"seq":64424515615,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5625040,"kb_used_data":4244484,"kb_used_omap":422,"kb_used_meta":1380121,"kb_avail":88746800,"statfs":{"total":96636764160,"available":90876723200,"internally_reserved":0,"allocated":4346351616,"data_stored":4336938095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":432402,"internal_metadata":1413244654},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":0.88400000000000001,"15min":0.97199999999999998},"min":{"1min":0.20200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":8.3789999999999996,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.96199999999999997,"15min":1.181},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1079999999999997,"5min":5.1470000000000002,"15min":8.1039999999999992},"last":0.247}]},{"osd":2,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.159,"15min":1.194},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.2759999999999998,"5min":7.2759999999999998,"15min":10.146000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.2130000000000001,"15min":1.2889999999999999},"min":{"1min":0.26600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.4770000000000003,"5min":13.676,"15min":13.676},"last":0.39200000000000002}]},{"osd":3,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.2929999999999999,"15min":1.2749999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.25},"max":{"1min":2.5680000000000001,"5min":13.721,"15min":13.721},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.0840000000000001,"15min":1.1870000000000001},"min":{"1min":0.37,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":6.2089999999999996,"5min":8.0570000000000004,"15min":10.037000000000001},"last":1.5880000000000001}]},{"osd":4,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.145,"15min":1.1919999999999999},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.0579999999999998,"5min":13.755000000000001,"15min":13.755000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.0389999999999999,"15min":0.93200000000000005},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.29199999999999998},"max":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":7.3540000000000001},"last":0.30299999999999999}]},{"osd":5,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":0.95499999999999996,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.32700000000000001},"max":{"1min":6.5090000000000003,"5min":6.5090000000000003,"15min":7.4349999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.0089999999999999,"15min":1.149},"min":{"1min":0.379,"5min":0.31,"15min":0.31},"max":{"1min":4.1589999999999998,"5min":5.101,"15min":8.1820000000000004},"last":0.58999999999999997}]},{"osd":6,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.25,"15min":1.3069999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.34300000000000003},"max":{"1min":6.202,"5min":13.819000000000001,"15min":13.819000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.1499999999999999,"15min":1.087},"min":{"1min":0.38,"5min":0.38,"15min":0.378},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":0.46300000000000002}]},{"osd":7,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.087,"15min":1.2030000000000001},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":4.2119999999999997,"5min":5.1760000000000002,"15min":8.2110000000000003},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.266,"15min":1.4059999999999999},"min":{"1min":0.39500000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.242,"5min":13.786,"15min":13.786},"last":0.44}]}]},{"osd":2,"up_from":14,"seq":60129548316,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5800052,"kb_used_data":4523496,"kb_used_omap":313,"kb_used_meta":1276230,"kb_avail":88571788,"statfs":{"total":96636764160,"available":90697510912,"internally_reserved":0,"allocated":4632059904,"data_stored":4622129985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":320629,"internal_metadata":1306860427},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":0.96099999999999997,"15min":1.0049999999999999},"min":{"1min":0.23400000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.8399999999999999,"5min":7.5919999999999996,"15min":8.1500000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3120000000000001,"15min":1.1439999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":10.301,"5min":10.301,"15min":10.301},"last":0.59499999999999997}]},{"osd":1,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0329999999999999,"15min":1.103},"min":{"1min":0.26600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":4.9039999999999999,"5min":6.8010000000000002,"15min":7.7320000000000002},"last":0.438},{"interface":"front","average":{"1min":1.181,"5min":1.333,"15min":1.27},"min":{"1min":0.28299999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":10.345000000000001,"5min":10.345000000000001,"15min":10.345000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":1.446,"15min":1.4199999999999999},"min":{"1min":0.26700000000000002,"5min":0.24299999999999999,"15min":0.23999999999999999},"max":{"1min":10.706,"5min":10.706,"15min":10.706},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.042,"15min":1.1990000000000001},"min":{"1min":0.36699999999999999,"5min":0.29699999999999999,"15min":0.26100000000000001},"max":{"1min":4.9340000000000002,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.58899999999999997}]},{"osd":4,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":0.999,"15min":1.0920000000000001},"min":{"1min":0.374,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":4.7990000000000004,"5min":6.1870000000000003,"15min":8.0860000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.29,"15min":1.159},"min":{"1min":0.375,"5min":0.314,"15min":0.314},"max":{"1min":4.1630000000000003,"5min":9.4399999999999995,"15min":9.4399999999999995},"last":0.66000000000000003}]},{"osd":5,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.383,"15min":1.2649999999999999},"min":{"1min":0.41599999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.9779999999999998,"5min":9.4890000000000008,"15min":9.4890000000000008},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.341,"15min":1.3759999999999999},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":7.8929999999999998,"5min":7.8929999999999998,"15min":10.366},"last":0.40899999999999997}]},{"osd":6,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":1.427,"15min":1.431},"min":{"1min":0.45900000000000002,"5min":0.39600000000000002,"15min":0.376},"max":{"1min":7.9770000000000003,"5min":7.9770000000000003,"15min":10.412000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.3,"5min":1.075,"15min":1.196},"min":{"1min":0.50800000000000001,"5min":0.36499999999999999,"15min":0.35999999999999999},"max":{"1min":4.8719999999999999,"5min":7.8959999999999999,"15min":8.0350000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Sun Dec 17 06:31:19 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.444,"15min":1.2909999999999999},"min":{"1min":0.44400000000000001,"5min":0.40100000000000002,"15min":0.37},"max":{"1min":5.8949999999999996,"5min":9.5489999999999995,"15min":9.5489999999999995},"last":0.86899999999999999},{"interface":"front","average":{"1min":2.4449999999999998,"5min":1.456,"15min":1.4670000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":7.9429999999999996,"5min":7.9429999999999996,"15min":10.446},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-17T06:32:19.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-17T06:32:20.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.046+0000 7f7d20d63640 1 Processor -- start 2023-12-17T06:32:20.054 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.046+0000 7f7d20d63640 1 -- start start 2023-12-17T06:32:20.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d20d63640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c15a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:20.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d20d63640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f7d1c159640 0x7f7d1c072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:20.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d20d63640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1c0b3b70 con 0x7f7d1c0b36c0 2023-12-17T06:32:20.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d1a59c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c15a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:20.056 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d1a59c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c15a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37242/0 (socket says 172.21.15.28:37242) 2023-12-17T06:32:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d1a59c640 1 -- 172.21.15.28:0/2587421582 learned_addr learned my addr 172.21.15.28:0/2587421582 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d13fff640 1 --2- 172.21.15.28:0/2587421582 >> v2:172.21.15.122:3300/0 conn(0x7f7d1c159640 0x7f7d1c072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d20d63640 1 -- 172.21.15.28:0/2587421582 --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d1c159ee0 con 0x7f7d1c159640 2023-12-17T06:32:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d20d63640 1 -- 172.21.15.28:0/2587421582 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c0b3e20 con 0x7f7d1c159260 2023-12-17T06:32:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d19d9b640 1 -- 172.21.15.28:0/2587421582 <== mon.1 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816322301 0 0) 0x7f7d1c0b3b70 con 0x7f7d1c0b36c0 2023-12-17T06:32:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d19d9b640 1 -- 172.21.15.28:0/2587421582 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d08005740 con 0x7f7d1c0b36c0 2023-12-17T06:32:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d13fff640 1 -- 172.21.15.28:0/2587421582 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 msgr2=0x7f7d1c15a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d13fff640 1 --2- 172.21.15.28:0/2587421582 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c15a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d13fff640 1 -- 172.21.15.28:0/2587421582 >> v1:172.21.15.122:6789/0 conn(0x7f7d1c0b36c0 legacy=0x7f7d1c06c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:20.058 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.050+0000 7f7d13fff640 1 -- 172.21.15.28:0/2587421582 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c0b4100 con 0x7f7d1c159640 2023-12-17T06:32:20.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d13fff640 1 --2- 172.21.15.28:0/2587421582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 0x7f7d1c072230 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7d040097c0 tx=0x7f7d0401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c881769436bb46 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:20.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d19d9b640 1 -- 172.21.15.28:0/2587421582 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040ab600 con 0x7f7d1c159640 2023-12-17T06:32:20.059 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d19d9b640 1 -- 172.21.15.28:0/2587421582 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d040a7050 con 0x7f7d1c159640 2023-12-17T06:32:20.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d19d9b640 1 -- 172.21.15.28:0/2587421582 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040ab9c0 con 0x7f7d1c159640 2023-12-17T06:32:20.060 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 -- 172.21.15.28:0/2587421582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 msgr2=0x7f7d1c072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:20.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 --2- 172.21.15.28:0/2587421582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 0x7f7d1c072230 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f7d040097c0 tx=0x7f7d0401c410 comp rx=0 tx=0).stop 2023-12-17T06:32:20.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 -- 172.21.15.28:0/2587421582 shutdown_connections 2023-12-17T06:32:20.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 --2- 172.21.15.28:0/2587421582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 0x7f7d1c072230 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 --2- 172.21.15.28:0/2587421582 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c15a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 -- 172.21.15.28:0/2587421582 >> 172.21.15.28:0/2587421582 conn(0x7f7d1c06a530 msgr2=0x7f7d1c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:20.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 -- 172.21.15.28:0/2587421582 shutdown_connections 2023-12-17T06:32:20.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 -- 172.21.15.28:0/2587421582 wait complete. 2023-12-17T06:32:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 Processor -- start 2023-12-17T06:32:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.054+0000 7f7d20d63640 1 -- start start 2023-12-17T06:32:20.064 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d20d63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d1c0b36c0 0x7f7d1c14cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d1a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d1c0b36c0 0x7f7d1c14cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d1a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d1c0b36c0 0x7f7d1c14cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36666/0 (socket says 172.21.15.28:36666) 2023-12-17T06:32:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d1a59c640 1 -- 172.21.15.28:0/3458054138 learned_addr learned my addr 172.21.15.28:0/3458054138 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d20d63640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c1546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d20d63640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 0x7f7d1c154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d13fff640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c1546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d20d63640 1 -- 172.21.15.28:0/3458054138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c14d840 con 0x7f7d1c0b36c0 2023-12-17T06:32:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d1ad9d640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 0x7f7d1c154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d20d63640 1 -- 172.21.15.28:0/3458054138 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c14db20 con 0x7f7d1c159640 2023-12-17T06:32:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d20d63640 1 -- 172.21.15.28:0/3458054138 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c14de00 con 0x7f7d1c159260 2023-12-17T06:32:20.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d13fff640 1 -- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 msgr2=0x7f7d1c154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:20.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d13fff640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 0x7f7d1c154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d13fff640 1 -- 172.21.15.28:0/3458054138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d1c0b36c0 msgr2=0x7f7d1c14cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:20.067 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d13fff640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d1c0b36c0 0x7f7d1c14cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d13fff640 1 -- 172.21.15.28:0/3458054138 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d04009420 con 0x7f7d1c159260 2023-12-17T06:32:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.058+0000 7f7d13fff640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c1546f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d040ab000 tx=0x7f7d040191b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d12ffd640 1 -- 172.21.15.28:0/3458054138 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040b4410 con 0x7f7d1c159260 2023-12-17T06:32:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d20d63640 1 -- 172.21.15.28:0/3458054138 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1c071180 con 0x7f7d1c159260 2023-12-17T06:32:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d20d63640 1 -- 172.21.15.28:0/3458054138 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1c14e3c0 con 0x7f7d1c159260 2023-12-17T06:32:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d12ffd640 1 -- 172.21.15.28:0/3458054138 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d040b4590 con 0x7f7d1c159260 2023-12-17T06:32:20.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d12ffd640 1 -- 172.21.15.28:0/3458054138 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040b4940 con 0x7f7d1c159260 2023-12-17T06:32:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d12ffd640 1 -- 172.21.15.28:0/3458054138 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7d040b2040 con 0x7f7d1c159260 2023-12-17T06:32:20.069 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d20d63640 1 -- 172.21.15.28:0/3458054138 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d00005260 con 0x7f7d1c159260 2023-12-17T06:32:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d12ffd640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7cfc066540 0x7f7cfc068a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d1a59c640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7cfc066540 0x7f7cfc068a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.062+0000 7f7d12ffd640 1 -- 172.21.15.28:0/3458054138 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7d0411da00 con 0x7f7d1c159260 2023-12-17T06:32:20.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.066+0000 7f7d1a59c640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7cfc066540 0x7f7cfc068a00 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0b1b20 tx=0x7f7d0c00c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:20.078 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.070+0000 7f7d12ffd640 1 -- 172.21.15.28:0/3458054138 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7d040eb2a0 con 0x7f7d1c159260 2023-12-17T06:32:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.358+0000 7f7d20d63640 1 -- 172.21.15.28:0/3458054138 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d00005510 con 0x7f7d1c159260 2023-12-17T06:32:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.358+0000 7f7d12ffd640 1 -- 172.21.15.28:0/3458054138 <== mon.2 v2:172.21.15.122:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+12972 (secure 0 0 0) 0x7f7d040ee8c0 con 0x7f7d1c159260 2023-12-17T06:32:20.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:20.369 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":39,"fsid":"fa65dfdd-b227-4a96-8047-b713453c357b","created":"2023-12-16T21:57:18.722917+0000","modified":"2023-12-17T06:32:12.758761+0000","last_up_change":"2023-12-16T21:57:30.460063+0000","last_in_change":"2023-12-16T21:57:21.083225+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:57:31.458376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"814fdfdf-9ed9-440c-814d-7d73b1a21072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6812","nonce":177892190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6816","nonce":177892190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6822","nonce":177892190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":177892190},{"type":"v1","addr":"172.21.15.28:6819","nonce":177892190}]},"public_addr":"172.21.15.28:6812/177892190","cluster_addr":"172.21.15.28:6816/177892190","heartbeat_back_addr":"172.21.15.28:6822/177892190","heartbeat_front_addr":"172.21.15.28:6819/177892190","state":["exists","up"]},{"osd":1,"uuid":"8f53d5ae-12c2-4944-9728-b31af8119226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6825","nonce":3615286709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6827","nonce":3615286709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6831","nonce":3615286709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3615286709},{"type":"v1","addr":"172.21.15.28:6829","nonce":3615286709}]},"public_addr":"172.21.15.28:6825/3615286709","cluster_addr":"172.21.15.28:6827/3615286709","heartbeat_back_addr":"172.21.15.28:6831/3615286709","heartbeat_front_addr":"172.21.15.28:6829/3615286709","state":["exists","up"]},{"osd":2,"uuid":"1804f849-d8f5-4fbb-a6e1-6b7572c80d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6807","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6809","nonce":451272560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6813","nonce":451272560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6823","nonce":451272560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":451272560},{"type":"v1","addr":"172.21.15.28:6818","nonce":451272560}]},"public_addr":"172.21.15.28:6809/451272560","cluster_addr":"172.21.15.28:6813/451272560","heartbeat_back_addr":"172.21.15.28:6823/451272560","heartbeat_front_addr":"172.21.15.28:6818/451272560","state":["exists","up"]},{"osd":3,"uuid":"98e0c424-17ea-4566-a959-48efa476f1a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6801","nonce":4021281394}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6803","nonce":4021281394}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6808","nonce":4021281394}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":4021281394},{"type":"v1","addr":"172.21.15.28:6805","nonce":4021281394}]},"public_addr":"172.21.15.28:6801/4021281394","cluster_addr":"172.21.15.28:6803/4021281394","heartbeat_back_addr":"172.21.15.28:6808/4021281394","heartbeat_front_addr":"172.21.15.28:6805/4021281394","state":["exists","up"]},{"osd":4,"uuid":"2585d9a5-cc84-4342-9684-4eda8e1d7513","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6801","nonce":1281108383}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6803","nonce":1281108383}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6807","nonce":1281108383}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1281108383},{"type":"v1","addr":"172.21.15.122:6805","nonce":1281108383}]},"public_addr":"172.21.15.122:6801/1281108383","cluster_addr":"172.21.15.122:6803/1281108383","heartbeat_back_addr":"172.21.15.122:6807/1281108383","heartbeat_front_addr":"172.21.15.122:6805/1281108383","state":["exists","up"]},{"osd":5,"uuid":"a2afbca9-85c5-4583-9cbf-33cbf0a54677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6817","nonce":1068560627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6819","nonce":1068560627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6823","nonce":1068560627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":1068560627},{"type":"v1","addr":"172.21.15.122:6821","nonce":1068560627}]},"public_addr":"172.21.15.122:6817/1068560627","cluster_addr":"172.21.15.122:6819/1068560627","heartbeat_back_addr":"172.21.15.122:6823/1068560627","heartbeat_front_addr":"172.21.15.122:6821/1068560627","state":["exists","up"]},{"osd":6,"uuid":"b7d9aa20-7c15-425a-922b-659558ce1ef0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6825","nonce":1879423542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6827","nonce":1879423542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6831","nonce":1879423542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1879423542},{"type":"v1","addr":"172.21.15.122:6829","nonce":1879423542}]},"public_addr":"172.21.15.122:6825/1879423542","cluster_addr":"172.21.15.122:6827/1879423542","heartbeat_back_addr":"172.21.15.122:6831/1879423542","heartbeat_front_addr":"172.21.15.122:6829/1879423542","state":["exists","up"]},{"osd":7,"uuid":"3160ca90-072b-47b7-9f5a-e9a306cdf4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6809","nonce":4051779013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6811","nonce":4051779013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6815","nonce":4051779013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4051779013},{"type":"v1","addr":"172.21.15.122:6813","nonce":4051779013}]},"public_addr":"172.21.15.122:6809/4051779013","cluster_addr":"172.21.15.122:6811/4051779013","heartbeat_back_addr":"172.21.15.122:6815/4051779013","heartbeat_front_addr":"172.21.15.122:6813/4051779013","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.485149+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.473085+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.335548+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.450729+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.498508+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.652614+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.605708+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:57:26.484236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/924398795":"2023-12-18T06:32:09.727580+0000","172.21.15.28:6832/924398795":"2023-12-18T06:32:09.727580+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-17T06:32:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 -- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7cfc066540 msgr2=0x7f7cfc068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7cfc066540 0x7f7cfc068a00 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0b1b20 tx=0x7f7d0c00c040 comp rx=0 tx=0).stop 2023-12-17T06:32:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 -- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 msgr2=0x7f7d1c1546f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c1546f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d040ab000 tx=0x7f7d040191b0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 -- 172.21.15.28:0/3458054138 shutdown_connections 2023-12-17T06:32:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f7cfc066540 0x7f7cfc068a00 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d1c159640 0x7f7d1c154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f7d1c159260 0x7f7d1c1546f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 --2- 172.21.15.28:0/3458054138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d1c0b36c0 0x7f7d1c14cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 -- 172.21.15.28:0/3458054138 >> 172.21.15.28:0/3458054138 conn(0x7f7d1c06a530 msgr2=0x7f7d1c15c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 -- 172.21.15.28:0/3458054138 shutdown_connections 2023-12-17T06:32:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:20.378+0000 7f7d10ff9640 1 -- 172.21.15.28:0/3458054138 wait complete. 2023-12-17T06:32:21.406 INFO:tasks.ceph:Scrubbing osd.0 2023-12-17T06:32:21.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-17T06:32:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a2684640 1 Processor -- start 2023-12-17T06:32:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a2684640 1 -- start start 2023-12-17T06:32:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a2684640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0cd270 0x7ff79c0d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a2684640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff79c09a0e0 0x7ff79c0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a2684640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79c0d1ec0 con 0x7ff79c0d1a40 2023-12-17T06:32:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a2684640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c06c180 con 0x7ff79c09a0e0 2023-12-17T06:32:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a2684640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c06c460 con 0x7ff79c0cd270 2023-12-17T06:32:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff79b7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff79c0d1a40 0x7ff79c0999d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46106/0 (socket says 172.21.15.28:46106) 2023-12-17T06:32:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff79b7fe640 1 -- 172.21.15.28:0/2264563552 learned_addr learned my addr 172.21.15.28:0/2264563552 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:21.546 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a0c21640 1 --2- 172.21.15.28:0/2264563552 >> v2:172.21.15.28:3300/0 conn(0x7ff79c09a0e0 0x7ff79c0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.547 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff79bfff640 1 --2- 172.21.15.28:0/2264563552 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0cd270 0x7ff79c0d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.549 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a0c21640 1 -- 172.21.15.28:0/2264563552 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0cd270 msgr2=0x7ff79c0d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a0c21640 1 --2- 172.21.15.28:0/2264563552 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0cd270 0x7ff79c0d0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a0c21640 1 -- 172.21.15.28:0/2264563552 >> v1:172.21.15.28:6789/0 conn(0x7ff79c0d1a40 legacy=0x7ff79c0999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.538+0000 7ff7a0c21640 1 -- 172.21.15.28:0/2264563552 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c0cf150 con 0x7ff79c09a0e0 2023-12-17T06:32:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a0c21640 1 --2- 172.21.15.28:0/2264563552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c09a0e0 0x7ff79c0cb570 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7ff794009d90 tx=0x7ff79401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe1f46d4f672c01 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff79affd640 1 -- 172.21.15.28:0/2264563552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7940a5a00 con 0x7ff79c09a0e0 2023-12-17T06:32:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff79affd640 1 -- 172.21.15.28:0/2264563552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7940a5b60 con 0x7ff79c09a0e0 2023-12-17T06:32:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff79affd640 1 -- 172.21.15.28:0/2264563552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7940ad440 con 0x7ff79c09a0e0 2023-12-17T06:32:21.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a2684640 1 -- 172.21.15.28:0/2264563552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c09a0e0 msgr2=0x7ff79c0cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.551 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a2684640 1 --2- 172.21.15.28:0/2264563552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c09a0e0 0x7ff79c0cb570 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7ff794009d90 tx=0x7ff79401c920 comp rx=0 tx=0).stop 2023-12-17T06:32:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a2684640 1 -- 172.21.15.28:0/2264563552 shutdown_connections 2023-12-17T06:32:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a2684640 1 --2- 172.21.15.28:0/2264563552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c09a0e0 0x7ff79c0cb570 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a2684640 1 --2- 172.21.15.28:0/2264563552 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0cd270 0x7ff79c0d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a2684640 1 -- 172.21.15.28:0/2264563552 >> 172.21.15.28:0/2264563552 conn(0x7ff79c0d0f20 msgr2=0x7ff79c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a2684640 1 -- 172.21.15.28:0/2264563552 shutdown_connections 2023-12-17T06:32:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.542+0000 7ff7a2684640 1 -- 172.21.15.28:0/2264563552 wait complete. 2023-12-17T06:32:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 Processor -- start 2023-12-17T06:32:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 -- start start 2023-12-17T06:32:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff79c09a0e0 0x7ff79c0b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c0cd270 0x7ff79c0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0d1a40 0x7ff79c06e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c156ec0 con 0x7ff79c0cd270 2023-12-17T06:32:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0b40e0 con 0x7ff79c09a0e0 2023-12-17T06:32:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c0cd270 0x7ff79c0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c0cd270 0x7ff79c0b6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36686/0 (socket says 172.21.15.28:36686) 2023-12-17T06:32:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 -- 172.21.15.28:0/61053694 learned_addr learned my addr 172.21.15.28:0/61053694 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c157110 con 0x7ff79c0d1a40 2023-12-17T06:32:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79bfff640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff79c09a0e0 0x7ff79c0b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a0c21640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0d1a40 0x7ff79c06e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 -- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0d1a40 msgr2=0x7ff79c06e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff79c0d1a40 0x7ff79c06e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 -- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff79c09a0e0 msgr2=0x7ff79c0b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff79c09a0e0 0x7ff79c0b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 -- 172.21.15.28:0/61053694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff794009a70 con 0x7ff79c0cd270 2023-12-17T06:32:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff79b7fe640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c0cd270 0x7ff79c0b6350 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7ff79000c530 tx=0x7ff79000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790005cf0 con 0x7ff79c0cd270 2023-12-17T06:32:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff790005e50 con 0x7ff79c0cd270 2023-12-17T06:32:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790007270 con 0x7ff79c0cd270 2023-12-17T06:32:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 -- 172.21.15.28:0/61053694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c1573f0 con 0x7ff79c0cd270 2023-12-17T06:32:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.546+0000 7ff7a2684640 1 -- 172.21.15.28:0/61053694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c0c6750 con 0x7ff79c0cd270 2023-12-17T06:32:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff774000fd0 con 0x7ff79c0cd270 2023-12-17T06:32:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7ff7900073d0 con 0x7ff79c0cd270 2023-12-17T06:32:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff798ff9640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff788066470 0x7ff788068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff790082fe0 con 0x7ff79c0cd270 2023-12-17T06:32:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff79bfff640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff788066470 0x7ff788068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff798ff9640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7ff78806c2d0 0x7ff78806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 --> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff78806eda0 con 0x7ff78806c2d0 2023-12-17T06:32:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7900855c0 con 0x7ff79c0cd270 2023-12-17T06:32:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff7a0c21640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7ff78806c2d0 0x7ff78806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff79bfff640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff788066470 0x7ff788068930 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009b60 tx=0x7ff78c009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff7a0c21640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7ff78806c2d0 0x7ff78806e6f0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.550+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 <== osd.0 v2:172.21.15.28:6810/177892190 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff78806eda0 con 0x7ff78806c2d0 2023-12-17T06:32:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.574+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 --> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff774002c60 con 0x7ff78806c2d0 2023-12-17T06:32:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff798ff9640 1 -- 172.21.15.28:0/61053694 <== osd.0 v2:172.21.15.28:6810/177892190 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff774002c60 con 0x7ff78806c2d0 2023-12-17T06:32:21.583 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-17T06:32:21.584 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-17T06:32:21.584 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-17T06:32:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7ff78806c2d0 msgr2=0x7ff78806e6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.28:6810/177892190,v1:172.21.15.28:6812/177892190] conn(0x7ff78806c2d0 0x7ff78806e6f0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff788066470 msgr2=0x7ff788068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff788066470 0x7ff788068930 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009b60 tx=0x7ff78c009190 comp rx=0 tx=0).stop 2023-12-17T06:32:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c0cd270 msgr2=0x7ff79c0b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 --2- 172.21.15.28:0/61053694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff79c0cd270 0x7ff79c0b6350 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7ff79000c530 tx=0x7ff79000c9f0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7a0c21640 1 -- 172.21.15.28:0/61053694 reap_dead start 2023-12-17T06:32:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 shutdown_connections 2023-12-17T06:32:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 >> 172.21.15.28:0/61053694 conn(0x7ff79c0d0f20 msgr2=0x7ff79c06fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 shutdown_connections 2023-12-17T06:32:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.578+0000 7ff7827fc640 1 -- 172.21.15.28:0/61053694 wait complete. 2023-12-17T06:32:21.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-17T06:32:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342af6e640 1 Processor -- start 2023-12-17T06:32:21.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342af6e640 1 -- start start 2023-12-17T06:32:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342af6e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f342413ded0 0x7f34240cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342af6e640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f342406e830 0x7f342406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342af6e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34240cfcf0 con 0x7f342413e2b0 2023-12-17T06:32:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f3428d0a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f342413ded0 0x7f34240cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342af6e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f342406f220 con 0x7f342413ded0 2023-12-17T06:32:21.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342af6e640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f342415b910 con 0x7f342406e830 2023-12-17T06:32:21.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f3428d0a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f342413ded0 0x7f34240cf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36692/0 (socket says 172.21.15.28:36692) 2023-12-17T06:32:21.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f3428d0a640 1 -- 172.21.15.28:0/2626428953 learned_addr learned my addr 172.21.15.28:0/2626428953 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:21.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342950b640 1 --2- 172.21.15.28:0/2626428953 >> v2:172.21.15.122:3300/0 conn(0x7f342406e830 0x7f342406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f34237fe640 1 -- 172.21.15.28:0/2626428953 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272688554 0 0) 0x7f34240cfcf0 con 0x7f342413e2b0 2023-12-17T06:32:21.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f34237fe640 1 -- 172.21.15.28:0/2626428953 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3414003530 con 0x7f342413e2b0 2023-12-17T06:32:21.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342950b640 1 -- 172.21.15.28:0/2626428953 >> v2:172.21.15.28:3300/0 conn(0x7f342413ded0 msgr2=0x7f34240cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:21.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342950b640 1 --2- 172.21.15.28:0/2626428953 >> v2:172.21.15.28:3300/0 conn(0x7f342413ded0 0x7f34240cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342950b640 1 -- 172.21.15.28:0/2626428953 >> v1:172.21.15.28:6789/0 conn(0x7f342413e2b0 legacy=0x7f34241411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.758+0000 7f342950b640 1 -- 172.21.15.28:0/2626428953 --> v2:172.21.15.122:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342415bbf0 con 0x7f342406e830 2023-12-17T06:32:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.762+0000 7f342950b640 1 --2- 172.21.15.28:0/2626428953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342406e830 0x7f342406ece0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f34240d1700 tx=0x7f341c01c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=999ca6ec10b56d54 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.762+0000 7f34237fe640 1 -- 172.21.15.28:0/2626428953 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c0a59c0 con 0x7f342406e830 2023-12-17T06:32:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.762+0000 7f34237fe640 1 -- 172.21.15.28:0/2626428953 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c0a5b20 con 0x7f342406e830 2023-12-17T06:32:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.762+0000 7f34237fe640 1 -- 172.21.15.28:0/2626428953 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c0ad440 con 0x7f342406e830 2023-12-17T06:32:21.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.762+0000 7f342af6e640 1 -- 172.21.15.28:0/2626428953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342406e830 msgr2=0x7f342406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.762+0000 7f342af6e640 1 --2- 172.21.15.28:0/2626428953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342406e830 0x7f342406ece0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f34240d1700 tx=0x7f341c01c900 comp rx=0 tx=0).stop 2023-12-17T06:32:21.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 -- 172.21.15.28:0/2626428953 shutdown_connections 2023-12-17T06:32:21.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 --2- 172.21.15.28:0/2626428953 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342406e830 0x7f342406ece0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 --2- 172.21.15.28:0/2626428953 >> v2:172.21.15.28:3300/0 conn(0x7f342413ded0 0x7f34240cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 -- 172.21.15.28:0/2626428953 >> 172.21.15.28:0/2626428953 conn(0x7f342406a930 msgr2=0x7f342406ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 -- 172.21.15.28:0/2626428953 shutdown_connections 2023-12-17T06:32:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 -- 172.21.15.28:0/2626428953 wait complete. 2023-12-17T06:32:21.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 Processor -- start 2023-12-17T06:32:21.773 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 -- start start 2023-12-17T06:32:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f342406e830 0x7f34241400f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342413ded0 0x7f3424143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342413e2b0 0x7f34241444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f3428d0a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f342406e830 0x7f34241400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f3428d0a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f342406e830 0x7f34241400f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37276/0 (socket says 172.21.15.28:37276) 2023-12-17T06:32:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f3428d0a640 1 -- 172.21.15.28:0/1450527014 learned_addr learned my addr 172.21.15.28:0/1450527014 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f3423fff640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342413ded0 0x7f3424143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342950b640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342413e2b0 0x7f34241444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34241406d0 con 0x7f342413e2b0 2023-12-17T06:32:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.766+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34241409b0 con 0x7f342413ded0 2023-12-17T06:32:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3424140c90 con 0x7f342406e830 2023-12-17T06:32:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342950b640 1 -- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f342406e830 msgr2=0x7f34241400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342950b640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f342406e830 0x7f34241400f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342950b640 1 -- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342413ded0 msgr2=0x7f3424143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342950b640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342413ded0 0x7f3424143f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342950b640 1 -- 172.21.15.28:0/1450527014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c009a70 con 0x7f342413e2b0 2023-12-17T06:32:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342950b640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342413e2b0 0x7f34241444a0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f341c0ac000 tx=0x7f341c0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f34217fa640 1 -- 172.21.15.28:0/1450527014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c0be070 con 0x7f342413e2b0 2023-12-17T06:32:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f34217fa640 1 -- 172.21.15.28:0/1450527014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c0adae0 con 0x7f342413e2b0 2023-12-17T06:32:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3424140f70 con 0x7f342413e2b0 2023-12-17T06:32:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3424157f40 con 0x7f342413e2b0 2023-12-17T06:32:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f34217fa640 1 -- 172.21.15.28:0/1450527014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c0b9450 con 0x7f342413e2b0 2023-12-17T06:32:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.770+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34240674d0 con 0x7f342413e2b0 2023-12-17T06:32:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.774+0000 7f34217fa640 1 -- 172.21.15.28:0/1450527014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f341c0b95b0 con 0x7f342413e2b0 2023-12-17T06:32:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.774+0000 7f34217fa640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3408066470 0x7f3408068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.774+0000 7f3428d0a640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3408066470 0x7f3408068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.774+0000 7f3428d0a640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3408066470 0x7f3408068930 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f34241f70a0 tx=0x7f3418009240 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.782+0000 7f34217fa640 1 -- 172.21.15.28:0/1450527014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f341c0f58d0 con 0x7f342413e2b0 2023-12-17T06:32:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:21.782+0000 7f34217fa640 1 -- 172.21.15.28:0/1450527014 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f341c0eba80 con 0x7f342413e2b0 2023-12-17T06:32:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.006+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f342415b5a0 con 0x7f3408066470 2023-12-17T06:32:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f34217fa640 1 -- 172.21.15.28:0/1450527014 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3418009900 con 0x7f3408066470 2023-12-17T06:32:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 0 to deep-scrub 2023-12-17T06:32:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3408066470 msgr2=0x7f3408068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3408066470 0x7f3408068930 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f34241f70a0 tx=0x7f3418009240 comp rx=0 tx=0).stop 2023-12-17T06:32:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342413e2b0 msgr2=0x7f34241444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342413e2b0 0x7f34241444a0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f341c0ac000 tx=0x7f341c0196a0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 shutdown_connections 2023-12-17T06:32:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f3408066470 0x7f3408068930 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342413e2b0 0x7f34241444a0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f342413ded0 0x7f3424143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 --2- 172.21.15.28:0/1450527014 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f342406e830 0x7f34241400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 >> 172.21.15.28:0/1450527014 conn(0x7f342406a930 msgr2=0x7f342415aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 shutdown_connections 2023-12-17T06:32:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.010+0000 7f342af6e640 1 -- 172.21.15.28:0/1450527014 wait complete. 2023-12-17T06:32:22.038 INFO:tasks.ceph:Scrubbing osd.1 2023-12-17T06:32:22.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-17T06:32:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677fa84640 1 Processor -- start 2023-12-17T06:32:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677fa84640 1 -- start start 2023-12-17T06:32:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677fa84640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f677815aad0 0x7f677815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677fa84640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f67780ce790 0x7f67780cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677fa84640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677813f5d0 0x7f677813fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677fa84640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677813ffa0 con 0x7f677815aad0 2023-12-17T06:32:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677fa84640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67780b2ed0 con 0x7f67780ce790 2023-12-17T06:32:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677fa84640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67780cf1e0 con 0x7f677813f5d0 2023-12-17T06:32:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d820640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f677815aad0 0x7f677815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d820640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f677815aad0 0x7f677815aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36700/0 (socket says 172.21.15.28:36700) 2023-12-17T06:32:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d820640 1 -- 172.21.15.28:0/567283100 learned_addr learned my addr 172.21.15.28:0/567283100 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677e021640 1 --2- 172.21.15.28:0/567283100 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677813f5d0 0x7f677813fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d01f640 1 --2- 172.21.15.28:0/567283100 >> v2:172.21.15.122:3300/0 conn(0x7f67780ce790 0x7f67780cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d820640 1 -- 172.21.15.28:0/567283100 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677813f5d0 msgr2=0x7f677813fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d820640 1 --2- 172.21.15.28:0/567283100 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677813f5d0 0x7f677813fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d820640 1 -- 172.21.15.28:0/567283100 >> v2:172.21.15.122:3300/0 conn(0x7f67780ce790 msgr2=0x7f67780cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d820640 1 --2- 172.21.15.28:0/567283100 >> v2:172.21.15.122:3300/0 conn(0x7f67780ce790 0x7f67780cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d820640 1 -- 172.21.15.28:0/567283100 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67780cf4c0 con 0x7f677815aad0 2023-12-17T06:32:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.222+0000 7f677d01f640 1 --2- 172.21.15.28:0/567283100 >> v2:172.21.15.122:3300/0 conn(0x7f67780ce790 0x7f67780cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677d820640 1 --2- 172.21.15.28:0/567283100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677815aad0 0x7f677815aeb0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f67780b2b10 tx=0x7f677401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc40395647eaa4cb server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677c81e640 1 -- 172.21.15.28:0/567283100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67740aa620 con 0x7f677815aad0 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677c81e640 1 -- 172.21.15.28:0/567283100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67740aa780 con 0x7f677815aad0 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677c81e640 1 -- 172.21.15.28:0/567283100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67740aab10 con 0x7f677815aad0 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 -- 172.21.15.28:0/567283100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677815aad0 msgr2=0x7f677815aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 --2- 172.21.15.28:0/567283100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677815aad0 0x7f677815aeb0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f67780b2b10 tx=0x7f677401c500 comp rx=0 tx=0).stop 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 -- 172.21.15.28:0/567283100 shutdown_connections 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 --2- 172.21.15.28:0/567283100 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677813f5d0 0x7f677813fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 --2- 172.21.15.28:0/567283100 >> v2:172.21.15.122:3300/0 conn(0x7f67780ce790 0x7f67780cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 --2- 172.21.15.28:0/567283100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677815aad0 0x7f677815aeb0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 -- 172.21.15.28:0/567283100 >> 172.21.15.28:0/567283100 conn(0x7f67780b1bd0 msgr2=0x7f67780b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 -- 172.21.15.28:0/567283100 shutdown_connections 2023-12-17T06:32:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.226+0000 7f677fa84640 1 -- 172.21.15.28:0/567283100 wait complete. 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 Processor -- start 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 -- start start 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67780ce790 0x7f677813bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677d820640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67780ce790 0x7f677813bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677d820640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67780ce790 0x7f677813bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36716/0 (socket says 172.21.15.28:36716) 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677d820640 1 -- 172.21.15.28:0/37936001 learned_addr learned my addr 172.21.15.28:0/37936001 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677813f5d0 0x7f67781522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677815aad0 0x7f6778159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778155db0 con 0x7f67780ce790 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6778156090 con 0x7f677813f5d0 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6778156370 con 0x7f677815aad0 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677d01f640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677813f5d0 0x7f67781522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677e021640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677815aad0 0x7f6778159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677e021640 1 -- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677813f5d0 msgr2=0x7f67781522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677e021640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f677813f5d0 0x7f67781522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677e021640 1 -- 172.21.15.28:0/37936001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67780ce790 msgr2=0x7f677813bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677e021640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67780ce790 0x7f677813bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677e021640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6768009420 con 0x7f677815aad0 2023-12-17T06:32:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677d820640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67780ce790 0x7f677813bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677e021640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677815aad0 0x7f6778159060 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f677000efc0 tx=0x7f677000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770009c00 con 0x7f677815aad0 2023-12-17T06:32:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677000f040 con 0x7f677815aad0 2023-12-17T06:32:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67700146c0 con 0x7f677815aad0 2023-12-17T06:32:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67740094d0 con 0x7f677815aad0 2023-12-17T06:32:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.230+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677813c090 con 0x7f677815aad0 2023-12-17T06:32:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6778155f20 con 0x7f677815aad0 2023-12-17T06:32:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 <== mon.2 v2:172.21.15.122:3301/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f6770014820 con 0x7f677815aad0 2023-12-17T06:32:22.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f676e7fc640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f675c066470 0x7f675c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 <== mon.2 v2:172.21.15.122:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f67700853a0 con 0x7f677815aad0 2023-12-17T06:32:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f676e7fc640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7f675c06c2d0 0x7f675c06e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f675c06eda0 con 0x7f675c06c2d0 2023-12-17T06:32:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f677d820640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f675c066470 0x7f675c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 <== mon.2 v2:172.21.15.122:3301/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6770085710 con 0x7f677815aad0 2023-12-17T06:32:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f677d01f640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7f675c06c2d0 0x7f675c06e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f677d01f640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7f675c06c2d0 0x7f675c06e6f0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f677d820640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f675c066470 0x7f675c068930 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f677401cda0 tx=0x7f67740b5040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.234+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 <== osd.1 v2:172.21.15.28:6824/3615286709 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f675c06eda0 con 0x7f675c06c2d0 2023-12-17T06:32:22.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 --> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f67780725c0 con 0x7f675c06c2d0 2023-12-17T06:32:22.267 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f676e7fc640 1 -- 172.21.15.28:0/37936001 <== osd.1 v2:172.21.15.28:6824/3615286709 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f67780725c0 con 0x7f675c06c2d0 2023-12-17T06:32:22.268 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-17T06:32:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-17T06:32:22.268 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-17T06:32:22.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7f675c06c2d0 msgr2=0x7f675c06e6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.28:6824/3615286709,v1:172.21.15.28:6825/3615286709] conn(0x7f675c06c2d0 0x7f675c06e6f0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.269 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f675c066470 msgr2=0x7f675c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f675c066470 0x7f675c068930 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f677401cda0 tx=0x7f67740b5040 comp rx=0 tx=0).stop 2023-12-17T06:32:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677815aad0 msgr2=0x7f6778159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 --2- 172.21.15.28:0/37936001 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f677815aad0 0x7f6778159060 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f677000efc0 tx=0x7f677000c490 comp rx=0 tx=0).stop 2023-12-17T06:32:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677e021640 1 -- 172.21.15.28:0/37936001 reap_dead start 2023-12-17T06:32:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 shutdown_connections 2023-12-17T06:32:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 >> 172.21.15.28:0/37936001 conn(0x7f67780b1bd0 msgr2=0x7f677815d040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 shutdown_connections 2023-12-17T06:32:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.262+0000 7f677fa84640 1 -- 172.21.15.28:0/37936001 wait complete. 2023-12-17T06:32:22.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-17T06:32:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa5033a0640 1 Processor -- start 2023-12-17T06:32:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa5033a0640 1 -- start start 2023-12-17T06:32:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa5033a0640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fa4fc0715b0 0x7fa4fc0cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa5033a0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4fc15a450 0x7fa4fc0ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa5033a0640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4fc072b80 con 0x7fa4fc071990 2023-12-17T06:32:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa5033a0640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc0b2fd0 con 0x7fa4fc15a450 2023-12-17T06:32:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa5033a0640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4fc0b32b0 con 0x7fa4fc0715b0 2023-12-17T06:32:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa50193d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa4fc071990 0x7fa4fc0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46120/0 (socket says 172.21.15.28:46120) 2023-12-17T06:32:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa50193d640 1 -- 172.21.15.28:0/4025708837 learned_addr learned my addr 172.21.15.28:0/4025708837 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa50093b640 1 --2- 172.21.15.28:0/4025708837 >> v2:172.21.15.28:3300/0 conn(0x7fa4fc15a450 0x7fa4fc0ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa50113c640 1 --2- 172.21.15.28:0/4025708837 >> v2:172.21.15.122:3300/0 conn(0x7fa4fc0715b0 0x7fa4fc0cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa50093b640 1 -- 172.21.15.28:0/4025708837 >> v2:172.21.15.122:3300/0 conn(0x7fa4fc0715b0 msgr2=0x7fa4fc0cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa50093b640 1 --2- 172.21.15.28:0/4025708837 >> v2:172.21.15.122:3300/0 conn(0x7fa4fc0715b0 0x7fa4fc0cf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa50093b640 1 -- 172.21.15.28:0/4025708837 >> v1:172.21.15.28:6789/0 conn(0x7fa4fc071990 legacy=0x7fa4fc0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.442+0000 7fa50093b640 1 -- 172.21.15.28:0/4025708837 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc0b3590 con 0x7fa4fc15a450 2023-12-17T06:32:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.446+0000 7fa50093b640 1 --2- 172.21.15.28:0/4025708837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc15a450 0x7fa4fc0ce840 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc074a30 tx=0x7fa4ec01c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b835f686dcdd4a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.446+0000 7fa4f3fff640 1 -- 172.21.15.28:0/4025708837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0ab600 con 0x7fa4fc15a450 2023-12-17T06:32:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.446+0000 7fa4f3fff640 1 -- 172.21.15.28:0/4025708837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec0a7050 con 0x7fa4fc15a450 2023-12-17T06:32:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.446+0000 7fa4f3fff640 1 -- 172.21.15.28:0/4025708837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0ab980 con 0x7fa4fc15a450 2023-12-17T06:32:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- 172.21.15.28:0/4025708837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc15a450 msgr2=0x7fa4fc0ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 --2- 172.21.15.28:0/4025708837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc15a450 0x7fa4fc0ce840 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc074a30 tx=0x7fa4ec01c2f0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- 172.21.15.28:0/4025708837 shutdown_connections 2023-12-17T06:32:22.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 --2- 172.21.15.28:0/4025708837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc15a450 0x7fa4fc0ce840 secure :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc074a30 tx=0x7fa4ec01c2f0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 --2- 172.21.15.28:0/4025708837 >> v2:172.21.15.122:3300/0 conn(0x7fa4fc0715b0 0x7fa4fc0cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- 172.21.15.28:0/4025708837 >> 172.21.15.28:0/4025708837 conn(0x7fa4fc06d530 msgr2=0x7fa4fc06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:22.456 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- 172.21.15.28:0/4025708837 shutdown_connections 2023-12-17T06:32:22.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- 172.21.15.28:0/4025708837 wait complete. 2023-12-17T06:32:22.457 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 Processor -- start 2023-12-17T06:32:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- start start 2023-12-17T06:32:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa4fc0715b0 0x7fa4fc139b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4fc071990 0x7fa4fc135d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc138d30 0x7fa4fc134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc13a540 con 0x7fa4fc138d30 2023-12-17T06:32:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc13d5a0 con 0x7fa4fc071990 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa5033a0640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc13da30 con 0x7fa4fc0715b0 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa50113c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa4fc0715b0 0x7fa4fc139b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa50193d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc138d30 0x7fa4fc134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa50113c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa4fc0715b0 0x7fa4fc139b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37292/0 (socket says 172.21.15.28:37292) 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.450+0000 7fa50113c640 1 -- 172.21.15.28:0/1193013605 learned_addr learned my addr 172.21.15.28:0/1193013605 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa50093b640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4fc071990 0x7fa4fc135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa50193d640 1 -- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa4fc0715b0 msgr2=0x7fa4fc139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa50193d640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa4fc0715b0 0x7fa4fc139b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa50193d640 1 -- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4fc071990 msgr2=0x7fa4fc135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa50193d640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4fc071990 0x7fa4fc135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa50193d640 1 -- 172.21.15.28:0/1193013605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec009420 con 0x7fa4fc138d30 2023-12-17T06:32:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa50193d640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc138d30 0x7fa4fc134fd0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400e4e0 tx=0x7fa4f400e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa4f1ffb640 1 -- 172.21.15.28:0/1193013605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4009500 con 0x7fa4fc138d30 2023-12-17T06:32:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc13a260 con 0x7fa4fc138d30 2023-12-17T06:32:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc13d230 con 0x7fa4fc138d30 2023-12-17T06:32:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa4f1ffb640 1 -- 172.21.15.28:0/1193013605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f4009660 con 0x7fa4fc138d30 2023-12-17T06:32:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa4f1ffb640 1 -- 172.21.15.28:0/1193013605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4009a30 con 0x7fa4fc138d30 2023-12-17T06:32:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa4f1ffb640 1 -- 172.21.15.28:0/1193013605 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fa4f4013480 con 0x7fa4fc138d30 2023-12-17T06:32:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.454+0000 7fa4f1ffb640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa4e0066470 0x7fa4e0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.458+0000 7fa4f1ffb640 1 -- 172.21.15.28:0/1193013605 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa4f4086cf0 con 0x7fa4fc138d30 2023-12-17T06:32:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.458+0000 7fa50113c640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa4e0066470 0x7fa4e0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.458+0000 7fa50113c640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa4e0066470 0x7fa4e0068930 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800b240 tx=0x7fa4f800b3a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.458+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc005260 con 0x7fa4fc138d30 2023-12-17T06:32:22.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.466+0000 7fa4f1ffb640 1 -- 172.21.15.28:0/1193013605 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fa4f4054500 con 0x7fa4fc138d30 2023-12-17T06:32:22.725 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.718+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4dc000bd0 con 0x7fa4e0066470 2023-12-17T06:32:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.718+0000 7fa4f1ffb640 1 -- 172.21.15.28:0/1193013605 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa4dc000bd0 con 0x7fa4e0066470 2023-12-17T06:32:22.726 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 1 to deep-scrub 2023-12-17T06:32:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa4e0066470 msgr2=0x7fa4e0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa4e0066470 0x7fa4e0068930 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800b240 tx=0x7fa4f800b3a0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc138d30 msgr2=0x7fa4fc134fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc138d30 0x7fa4fc134fd0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400e4e0 tx=0x7fa4f400e9a0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 shutdown_connections 2023-12-17T06:32:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fa4e0066470 0x7fa4e0068930 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4fc138d30 0x7fa4fc134fd0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4fc071990 0x7fa4fc135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 --2- 172.21.15.28:0/1193013605 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fa4fc0715b0 0x7fa4fc139b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 >> 172.21.15.28:0/1193013605 conn(0x7fa4fc06d530 msgr2=0x7fa4fc06a530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 shutdown_connections 2023-12-17T06:32:22.729 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.722+0000 7fa5033a0640 1 -- 172.21.15.28:0/1193013605 wait complete. 2023-12-17T06:32:22.743 INFO:tasks.ceph:Scrubbing osd.2 2023-12-17T06:32:22.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-17T06:32:22.854 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.846+0000 7f5c91473640 1 Processor -- start 2023-12-17T06:32:22.854 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.846+0000 7f5c91473640 1 -- start start 2023-12-17T06:32:22.855 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.846+0000 7f5c91473640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f5c8c09a0e0 0x7f5c8c0cb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.846+0000 7f5c91473640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5c8c0b25e0 0x7f5c8c0cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0b1ae0 con 0x7f5c8c15f7b0 2023-12-17T06:32:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c0b0150 con 0x7f5c8c0b25e0 2023-12-17T06:32:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c0b1490 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8affd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5c8c15f7b0 0x7f5c8c0999d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46136/0 (socket says 172.21.15.28:46136) 2023-12-17T06:32:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8b7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5c8c0b25e0 0x7f5c8c0cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.856 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8affd640 1 -- 172.21.15.28:0/1348827171 learned_addr learned my addr 172.21.15.28:0/1348827171 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8a7fc640 1 --2- 172.21.15.28:0/1348827171 >> v2:172.21.15.122:3300/0 conn(0x7f5c8c09a0e0 0x7f5c8c0cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8b7fe640 1 -- 172.21.15.28:0/1348827171 >> v2:172.21.15.122:3300/0 conn(0x7f5c8c09a0e0 msgr2=0x7f5c8c0cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8b7fe640 1 --2- 172.21.15.28:0/1348827171 >> v2:172.21.15.122:3300/0 conn(0x7f5c8c09a0e0 0x7f5c8c0cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8b7fe640 1 -- 172.21.15.28:0/1348827171 >> v1:172.21.15.28:6789/0 conn(0x7f5c8c15f7b0 legacy=0x7f5c8c0999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c89ffb640 1 -- 172.21.15.28:0/1348827171 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181358208 0 0) 0x7f5c8c0b1ae0 con 0x7f5c8c15f7b0 2023-12-17T06:32:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8b7fe640 1 -- 172.21.15.28:0/1348827171 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0b1770 con 0x7f5c8c0b25e0 2023-12-17T06:32:22.858 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c8b7fe640 1 --2- 172.21.15.28:0/1348827171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c0b25e0 0x7f5c8c0cbe40 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c0d0c50 tx=0x7f5c8401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe80fb91921f182 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c89ffb640 1 -- 172.21.15.28:0/1348827171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c840a58e0 con 0x7f5c8c0b25e0 2023-12-17T06:32:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c89ffb640 1 -- 172.21.15.28:0/1348827171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c840a5a40 con 0x7f5c8c0b25e0 2023-12-17T06:32:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c89ffb640 1 -- 172.21.15.28:0/1348827171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c840a5dd0 con 0x7f5c8c0b25e0 2023-12-17T06:32:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 -- 172.21.15.28:0/1348827171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c0b25e0 msgr2=0x7f5c8c0cbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 --2- 172.21.15.28:0/1348827171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c0b25e0 0x7f5c8c0cbe40 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c0d0c50 tx=0x7f5c8401c910 comp rx=0 tx=0).stop 2023-12-17T06:32:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 -- 172.21.15.28:0/1348827171 shutdown_connections 2023-12-17T06:32:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 --2- 172.21.15.28:0/1348827171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c0b25e0 0x7f5c8c0cbe40 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 --2- 172.21.15.28:0/1348827171 >> v2:172.21.15.122:3300/0 conn(0x7f5c8c09a0e0 0x7f5c8c0cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 -- 172.21.15.28:0/1348827171 >> 172.21.15.28:0/1348827171 conn(0x7f5c8c0d0f20 msgr2=0x7f5c8c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 -- 172.21.15.28:0/1348827171 shutdown_connections 2023-12-17T06:32:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.850+0000 7f5c91473640 1 -- 172.21.15.28:0/1348827171 wait complete. 2023-12-17T06:32:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 Processor -- start 2023-12-17T06:32:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 -- start start 2023-12-17T06:32:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c09a0e0 0x7f5c8c1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5c8c0b25e0 0x7f5c8c148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c8c15f7b0 0x7f5c8c06f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c158de0 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c1485c0 con 0x7f5c8c15f7b0 2023-12-17T06:32:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c0c8a00 con 0x7f5c8c0b25e0 2023-12-17T06:32:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c09a0e0 0x7f5c8c1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c09a0e0 0x7f5c8c1588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36744/0 (socket says 172.21.15.28:36744) 2023-12-17T06:32:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 -- 172.21.15.28:0/3473897401 learned_addr learned my addr 172.21.15.28:0/3473897401 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8a7fc640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5c8c0b25e0 0x7f5c8c148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8b7fe640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c8c15f7b0 0x7f5c8c06f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 -- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5c8c0b25e0 msgr2=0x7f5c8c148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f5c8c0b25e0 0x7f5c8c148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 -- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c8c15f7b0 msgr2=0x7f5c8c06f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c8c15f7b0 0x7f5c8c06f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 -- 172.21.15.28:0/3473897401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c84009a70 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c8affd640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c09a0e0 0x7f5c8c1588a0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00c480 tx=0x7f5c7c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c0074e0 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c007660 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c0c8ce0 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c004a30 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c0ca9a0 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.854+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5c8c0674d0 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f5c7c004b90 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c737fe640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5c78066470 0x7f5c78068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5c7c0833a0 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c737fe640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7f5c7806c2d0 0x7f5c7806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 --> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c7806eda0 con 0x7f5c7806c2d0 2023-12-17T06:32:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c7c083710 con 0x7f5c8c09a0e0 2023-12-17T06:32:22.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c8a7fc640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5c78066470 0x7f5c78068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c8b7fe640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7f5c7806c2d0 0x7f5c7806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:22.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c8b7fe640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7f5c7806c2d0 0x7f5c7806e6f0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c8a7fc640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5c78066470 0x7f5c78068930 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c1491c0 tx=0x7f5c80008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:22.868 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.858+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 <== osd.2 v2:172.21.15.28:6807/451272560 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5c7806eda0 con 0x7f5c7806c2d0 2023-12-17T06:32:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.870+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 --> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5c8c072800 con 0x7f5c7806c2d0 2023-12-17T06:32:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.870+0000 7f5c737fe640 1 -- 172.21.15.28:0/3473897401 <== osd.2 v2:172.21.15.28:6807/451272560 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5c8c072800 con 0x7f5c7806c2d0 2023-12-17T06:32:22.879 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-17T06:32:22.879 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-17T06:32:22.879 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-17T06:32:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.870+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7f5c7806c2d0 msgr2=0x7f5c7806e6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.870+0000 7f5c91473640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.28:6807/451272560,v1:172.21.15.28:6809/451272560] conn(0x7f5c7806c2d0 0x7f5c7806e6f0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5c78066470 msgr2=0x7f5c78068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c91473640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5c78066470 0x7f5c78068930 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c1491c0 tx=0x7f5c80008040 comp rx=0 tx=0).stop 2023-12-17T06:32:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c09a0e0 msgr2=0x7f5c8c1588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c91473640 1 --2- 172.21.15.28:0/3473897401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c8c09a0e0 0x7f5c8c1588a0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00c480 tx=0x7f5c7c00c940 comp rx=0 tx=0).stop 2023-12-17T06:32:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c8b7fe640 1 -- 172.21.15.28:0/3473897401 reap_dead start 2023-12-17T06:32:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 shutdown_connections 2023-12-17T06:32:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 >> 172.21.15.28:0/3473897401 conn(0x7f5c8c0d0f20 msgr2=0x7f5c8c0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 shutdown_connections 2023-12-17T06:32:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:22.874+0000 7f5c91473640 1 -- 172.21.15.28:0/3473897401 wait complete. 2023-12-17T06:32:22.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-17T06:32:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485e997640 1 Processor -- start 2023-12-17T06:32:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485e997640 1 -- start start 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485e997640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f485806c1f0 0x7f485815b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485e997640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48580ccc50 0x7f485815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485e997640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580cde50 con 0x7f48580cc870 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485e997640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485815c660 con 0x7f48580ccc50 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485e997640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485815c380 con 0x7f485806c1f0 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485cf34640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48580ccc50 0x7f485815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485cf34640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48580ccc50 0x7f485815bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36758/0 (socket says 172.21.15.28:36758) 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485cf34640 1 -- 172.21.15.28:0/1667209518 learned_addr learned my addr 172.21.15.28:0/1667209518 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f48577fe640 1 --2- 172.21.15.28:0/1667209518 >> v2:172.21.15.122:3300/0 conn(0x7f485806c1f0 0x7f485815b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485cf34640 1 -- 172.21.15.28:0/1667209518 >> v2:172.21.15.122:3300/0 conn(0x7f485806c1f0 msgr2=0x7f485815b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485cf34640 1 --2- 172.21.15.28:0/1667209518 >> v2:172.21.15.122:3300/0 conn(0x7f485806c1f0 0x7f485815b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485cf34640 1 -- 172.21.15.28:0/1667209518 >> v1:172.21.15.122:6789/0 conn(0x7f48580cc870 legacy=0x7f485806bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f485cf34640 1 -- 172.21.15.28:0/1667209518 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48580cd800 con 0x7f48580ccc50 2023-12-17T06:32:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.014+0000 7f48577fe640 1 --2- 172.21.15.28:0/1667209518 >> v2:172.21.15.122:3300/0 conn(0x7f485806c1f0 0x7f485815b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-17T06:32:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485cf34640 1 --2- 172.21.15.28:0/1667209518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580ccc50 0x7f485815bb40 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f48581526f0 tx=0x7f485001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1301790d3f9cc73 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f4856ffd640 1 -- 172.21.15.28:0/1667209518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48500a5770 con 0x7f48580ccc50 2023-12-17T06:32:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f4856ffd640 1 -- 172.21.15.28:0/1667209518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48500a58d0 con 0x7f48580ccc50 2023-12-17T06:32:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f4856ffd640 1 -- 172.21.15.28:0/1667209518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48500a5c60 con 0x7f48580ccc50 2023-12-17T06:32:23.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485e997640 1 -- 172.21.15.28:0/1667209518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580ccc50 msgr2=0x7f485815bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485e997640 1 --2- 172.21.15.28:0/1667209518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580ccc50 0x7f485815bb40 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f48581526f0 tx=0x7f485001c9a0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485e997640 1 -- 172.21.15.28:0/1667209518 shutdown_connections 2023-12-17T06:32:23.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485e997640 1 --2- 172.21.15.28:0/1667209518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580ccc50 0x7f485815bb40 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485e997640 1 --2- 172.21.15.28:0/1667209518 >> v2:172.21.15.122:3300/0 conn(0x7f485806c1f0 0x7f485815b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485e997640 1 -- 172.21.15.28:0/1667209518 >> 172.21.15.28:0/1667209518 conn(0x7f4858066320 msgr2=0x7f4858066730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.026 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485e997640 1 -- 172.21.15.28:0/1667209518 shutdown_connections 2023-12-17T06:32:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.018+0000 7f485e997640 1 -- 172.21.15.28:0/1667209518 wait complete. 2023-12-17T06:32:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 Processor -- start 2023-12-17T06:32:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 -- start start 2023-12-17T06:32:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485806c1f0 0x7f48581484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580cc870 0x7f4858155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48580ccc50 0x7f48581561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4858148a20 con 0x7f48580cc870 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48581f51f0 con 0x7f485806c1f0 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4858136bf0 con 0x7f48580ccc50 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580cc870 0x7f4858155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580cc870 0x7f4858155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36766/0 (socket says 172.21.15.28:36766) 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 -- 172.21.15.28:0/1073842449 learned_addr learned my addr 172.21.15.28:0/1073842449 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f4857fff640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485806c1f0 0x7f48581484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485cf34640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48580ccc50 0x7f48581561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 -- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48580ccc50 msgr2=0x7f48581561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48580ccc50 0x7f48581561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 -- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485806c1f0 msgr2=0x7f48581484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485806c1f0 0x7f48581484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 -- 172.21.15.28:0/1073842449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4850009b30 con 0x7f48580cc870 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f48577fe640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580cc870 0x7f4858155c80 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f484c00d800 tx=0x7f484c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f4854ff9640 1 -- 172.21.15.28:0/1073842449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c00b6c0 con 0x7f48580cc870 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f4854ff9640 1 -- 172.21.15.28:0/1073842449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c00b820 con 0x7f48580cc870 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4858136ed0 con 0x7f48580cc870 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f4854ff9640 1 -- 172.21.15.28:0/1073842449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c00bbd0 con 0x7f48580cc870 2023-12-17T06:32:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4858137490 con 0x7f48580cc870 2023-12-17T06:32:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4858073d20 con 0x7f48580cc870 2023-12-17T06:32:23.032 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.022+0000 7f4854ff9640 1 -- 172.21.15.28:0/1073842449 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f484c00bd30 con 0x7f48580cc870 2023-12-17T06:32:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.026+0000 7f4854ff9640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4844066470 0x7f4844068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.026+0000 7f4857fff640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4844066470 0x7f4844068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.026+0000 7f4857fff640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4844066470 0x7f4844068930 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f485806bf30 tx=0x7f4848009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.026+0000 7f4854ff9640 1 -- 172.21.15.28:0/1073842449 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f484c05b700 con 0x7f48580cc870 2023-12-17T06:32:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.030+0000 7f4854ff9640 1 -- 172.21.15.28:0/1073842449 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f484c051800 con 0x7f48580cc870 2023-12-17T06:32:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.182+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f48580674d0 con 0x7f4844066470 2023-12-17T06:32:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.182+0000 7f4854ff9640 1 -- 172.21.15.28:0/1073842449 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f48580674d0 con 0x7f4844066470 2023-12-17T06:32:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 2 to deep-scrub 2023-12-17T06:32:23.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4844066470 msgr2=0x7f4844068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4844066470 0x7f4844068930 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f485806bf30 tx=0x7f4848009040 comp rx=0 tx=0).stop 2023-12-17T06:32:23.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580cc870 msgr2=0x7f4858155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580cc870 0x7f4858155c80 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f484c00d800 tx=0x7f484c00dcc0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.192 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 shutdown_connections 2023-12-17T06:32:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f4844066470 0x7f4844068930 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f48580ccc50 0x7f48581561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48580cc870 0x7f4858155c80 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 --2- 172.21.15.28:0/1073842449 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f485806c1f0 0x7f48581484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 >> 172.21.15.28:0/1073842449 conn(0x7f4858066320 msgr2=0x7f4858073460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 shutdown_connections 2023-12-17T06:32:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.186+0000 7f485e997640 1 -- 172.21.15.28:0/1073842449 wait complete. 2023-12-17T06:32:23.207 INFO:tasks.ceph:Scrubbing osd.3 2023-12-17T06:32:23.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-17T06:32:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af8256640 1 Processor -- start 2023-12-17T06:32:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af8256640 1 -- start start 2023-12-17T06:32:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af8256640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f1af0146cb0 0x7f1af0160e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af8256640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1af0161370 0x7f1af00b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af8256640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af0161db0 con 0x7f1af01468d0 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af8256640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af015d7f0 con 0x7f1af0161370 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af8256640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af01617f0 con 0x7f1af0146cb0 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af5ff2640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1af01468d0 0x7f1af015e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46150/0 (socket says 172.21.15.28:46150) 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af5ff2640 1 -- 172.21.15.28:0/3217984833 learned_addr learned my addr 172.21.15.28:0/3217984833 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af67f3640 1 --2- 172.21.15.28:0/3217984833 >> v2:172.21.15.28:3300/0 conn(0x7f1af0161370 0x7f1af00b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af57f1640 1 --2- 172.21.15.28:0/3217984833 >> v2:172.21.15.122:3300/0 conn(0x7f1af0146cb0 0x7f1af0160e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af4ff0640 1 -- 172.21.15.28:0/3217984833 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133447359 0 0) 0x7f1af0161db0 con 0x7f1af01468d0 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af4ff0640 1 -- 172.21.15.28:0/3217984833 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1adc003530 con 0x7f1af01468d0 2023-12-17T06:32:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af67f3640 1 -- 172.21.15.28:0/3217984833 >> v2:172.21.15.122:3300/0 conn(0x7f1af0146cb0 msgr2=0x7f1af0160e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af67f3640 1 --2- 172.21.15.28:0/3217984833 >> v2:172.21.15.122:3300/0 conn(0x7f1af0146cb0 0x7f1af0160e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af67f3640 1 -- 172.21.15.28:0/3217984833 >> v1:172.21.15.28:6789/0 conn(0x7f1af01468d0 legacy=0x7f1af015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.314+0000 7f1af67f3640 1 -- 172.21.15.28:0/3217984833 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af0161ad0 con 0x7f1af0161370 2023-12-17T06:32:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af67f3640 1 --2- 172.21.15.28:0/3217984833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0161370 0x7f1af00b2fe0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f1af00b09d0 tx=0x7f1ae801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e640064b51bdf0fe server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af4ff0640 1 -- 172.21.15.28:0/3217984833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae80ab600 con 0x7f1af0161370 2023-12-17T06:32:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af4ff0640 1 -- 172.21.15.28:0/3217984833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae80a7050 con 0x7f1af0161370 2023-12-17T06:32:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af4ff0640 1 -- 172.21.15.28:0/3217984833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae80ab980 con 0x7f1af0161370 2023-12-17T06:32:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 -- 172.21.15.28:0/3217984833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0161370 msgr2=0x7f1af00b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 --2- 172.21.15.28:0/3217984833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0161370 0x7f1af00b2fe0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f1af00b09d0 tx=0x7f1ae801c3c0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 -- 172.21.15.28:0/3217984833 shutdown_connections 2023-12-17T06:32:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 --2- 172.21.15.28:0/3217984833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0161370 0x7f1af00b2fe0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 --2- 172.21.15.28:0/3217984833 >> v2:172.21.15.122:3300/0 conn(0x7f1af0146cb0 0x7f1af0160e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 -- 172.21.15.28:0/3217984833 >> 172.21.15.28:0/3217984833 conn(0x7f1af006a930 msgr2=0x7f1af006ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 -- 172.21.15.28:0/3217984833 shutdown_connections 2023-12-17T06:32:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 -- 172.21.15.28:0/3217984833 wait complete. 2023-12-17T06:32:23.327 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.318+0000 7f1af8256640 1 Processor -- start 2023-12-17T06:32:23.327 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 -- start start 2023-12-17T06:32:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af01468d0 0x7f1af015bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0146cb0 0x7f1af0143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1af0161370 0x7f1af0144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af013df60 con 0x7f1af0146cb0 2023-12-17T06:32:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af015c220 con 0x7f1af01468d0 2023-12-17T06:32:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af013d370 con 0x7f1af0161370 2023-12-17T06:32:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0146cb0 0x7f1af0143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0146cb0 0x7f1af0143b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36786/0 (socket says 172.21.15.28:36786) 2023-12-17T06:32:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 -- 172.21.15.28:0/2261148666 learned_addr learned my addr 172.21.15.28:0/2261148666 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af5ff2640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af01468d0 0x7f1af015bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af67f3640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1af0161370 0x7f1af0144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 -- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1af0161370 msgr2=0x7f1af0144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f1af0161370 0x7f1af0144040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 -- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af01468d0 msgr2=0x7f1af015bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1af01468d0 0x7f1af015bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 -- 172.21.15.28:0/2261148666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae8009420 con 0x7f1af0146cb0 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af57f1640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0146cb0 0x7f1af0143b00 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000e900 tx=0x7f1ae000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0009690 con 0x7f1af0146cb0 2023-12-17T06:32:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae000ca00 con 0x7f1af0146cb0 2023-12-17T06:32:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af013d5c0 con 0x7f1af0146cb0 2023-12-17T06:32:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000cdb0 con 0x7f1af0146cb0 2023-12-17T06:32:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af013db80 con 0x7f1af0146cb0 2023-12-17T06:32:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1af00b4240 con 0x7f1af0146cb0 2023-12-17T06:32:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f1ae0013470 con 0x7f1af0146cb0 2023-12-17T06:32:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.322+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1ad4066470 0x7f1ad4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1ae0084080 con 0x7f1af0146cb0 2023-12-17T06:32:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1ae67fc640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f1ad406c2d0 0x7f1ad406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 --> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ad406eda0 con 0x7f1ad406c2d0 2023-12-17T06:32:23.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ae00844d0 con 0x7f1af0146cb0 2023-12-17T06:32:23.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1af67f3640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f1ad406c2d0 0x7f1ad406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1af5ff2640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1ad4066470 0x7f1ad4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1af67f3640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f1ad406c2d0 0x7f1ad406e6f0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1af5ff2640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1ad4066470 0x7f1ad4068930 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1aec001570 tx=0x7f1aec001a00 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.334 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.326+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 <== osd.3 v2:172.21.15.28:6800/4021281394 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1ad406eda0 con 0x7f1ad406c2d0 2023-12-17T06:32:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 --> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1af0068070 con 0x7f1ad406c2d0 2023-12-17T06:32:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1ae67fc640 1 -- 172.21.15.28:0/2261148666 <== osd.3 v2:172.21.15.28:6800/4021281394 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1af0068070 con 0x7f1ad406c2d0 2023-12-17T06:32:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-17T06:32:23.346 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-17T06:32:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-17T06:32:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f1ad406c2d0 msgr2=0x7f1ad406e6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.28:6800/4021281394,v1:172.21.15.28:6801/4021281394] conn(0x7f1ad406c2d0 0x7f1ad406e6f0 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1ad4066470 msgr2=0x7f1ad4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f1ad4066470 0x7f1ad4068930 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1aec001570 tx=0x7f1aec001a00 comp rx=0 tx=0).stop 2023-12-17T06:32:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0146cb0 msgr2=0x7f1af0143b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 --2- 172.21.15.28:0/2261148666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af0146cb0 0x7f1af0143b00 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000e900 tx=0x7f1ae000edc0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af67f3640 1 -- 172.21.15.28:0/2261148666 reap_dead start 2023-12-17T06:32:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 shutdown_connections 2023-12-17T06:32:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 >> 172.21.15.28:0/2261148666 conn(0x7f1af006a930 msgr2=0x7f1af00d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 shutdown_connections 2023-12-17T06:32:23.348 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.338+0000 7f1af8256640 1 -- 172.21.15.28:0/2261148666 wait complete. 2023-12-17T06:32:23.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-17T06:32:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.458+0000 7fc833bca640 1 Processor -- start 2023-12-17T06:32:23.467 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.458+0000 7fc833bca640 1 -- start start 2023-12-17T06:32:23.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc833bca640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc82c15d3d0 0x7fc82c0ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.468 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc833bca640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc82c0af0a0 0x7fc82c06e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc833bca640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c06a530 0x7fc82c06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc833bca640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc82c15d7b0 con 0x7fc82c15d3d0 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc833bca640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc82c06af00 con 0x7fc82c0af0a0 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc833bca640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0af510 con 0x7fc82c06a530 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc82c15d3d0 0x7fc82c0ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc82c15d3d0 0x7fc82c0ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36792/0 (socket says 172.21.15.28:36792) 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 -- 172.21.15.28:0/920808357 learned_addr learned my addr 172.21.15.28:0/920808357 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831165640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fc82c0af0a0 0x7fc82c06e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc832167640 1 --2- 172.21.15.28:0/920808357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c06a530 0x7fc82c06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 -- 172.21.15.28:0/920808357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c06a530 msgr2=0x7fc82c06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 --2- 172.21.15.28:0/920808357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c06a530 0x7fc82c06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 -- 172.21.15.28:0/920808357 >> v2:172.21.15.122:3300/0 conn(0x7fc82c0af0a0 msgr2=0x7fc82c06e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 --2- 172.21.15.28:0/920808357 >> v2:172.21.15.122:3300/0 conn(0x7fc82c0af0a0 0x7fc82c06e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 -- 172.21.15.28:0/920808357 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82c0ad840 con 0x7fc82c15d3d0 2023-12-17T06:32:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc831966640 1 --2- 172.21.15.28:0/920808357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ae8a0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fc828009870 tx=0x7fc82801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a240b78504a0cf11 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc830964640 1 -- 172.21.15.28:0/920808357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8280aa5f0 con 0x7fc82c15d3d0 2023-12-17T06:32:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc830964640 1 -- 172.21.15.28:0/920808357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8280aa750 con 0x7fc82c15d3d0 2023-12-17T06:32:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.462+0000 7fc830964640 1 -- 172.21.15.28:0/920808357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8280aaae0 con 0x7fc82c15d3d0 2023-12-17T06:32:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- 172.21.15.28:0/920808357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 msgr2=0x7fc82c0ae8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 --2- 172.21.15.28:0/920808357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ae8a0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fc828009870 tx=0x7fc82801c4d0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- 172.21.15.28:0/920808357 shutdown_connections 2023-12-17T06:32:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 --2- 172.21.15.28:0/920808357 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c06a530 0x7fc82c06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 --2- 172.21.15.28:0/920808357 >> v2:172.21.15.122:3300/0 conn(0x7fc82c0af0a0 0x7fc82c06e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 --2- 172.21.15.28:0/920808357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ae8a0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- 172.21.15.28:0/920808357 >> 172.21.15.28:0/920808357 conn(0x7fc82c06d530 msgr2=0x7fc82c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- 172.21.15.28:0/920808357 shutdown_connections 2023-12-17T06:32:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- 172.21.15.28:0/920808357 wait complete. 2023-12-17T06:32:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 Processor -- start 2023-12-17T06:32:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- start start 2023-12-17T06:32:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc82c06a530 0x7fc82c0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c0af0a0 0x7fc82c1547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c14a0e0 con 0x7fc82c15d3d0 2023-12-17T06:32:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0af890 con 0x7fc82c06a530 2023-12-17T06:32:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c0afb70 con 0x7fc82c0af0a0 2023-12-17T06:32:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36796/0 (socket says 172.21.15.28:36796) 2023-12-17T06:32:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 -- 172.21.15.28:0/1010211278 learned_addr learned my addr 172.21.15.28:0/1010211278 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc831966640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc82c06a530 0x7fc82c0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc831165640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c0af0a0 0x7fc82c1547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 -- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c0af0a0 msgr2=0x7fc82c1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c0af0a0 0x7fc82c1547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 -- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc82c06a530 msgr2=0x7fc82c0cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc82c06a530 0x7fc82c0cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 -- 172.21.15.28:0/1010211278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8280094d0 con 0x7fc82c15d3d0 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc832167640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ceb90 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fc82400efc0 tx=0x7fc82400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc8227fc640 1 -- 172.21.15.28:0/1010211278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824014480 con 0x7fc82c15d3d0 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc8227fc640 1 -- 172.21.15.28:0/1010211278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82400f040 con 0x7fc82c15d3d0 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82c0afe50 con 0x7fc82c15d3d0 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c154cf0 con 0x7fc82c15d3d0 2023-12-17T06:32:23.477 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc8227fc640 1 -- 172.21.15.28:0/1010211278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824014820 con 0x7fc82c15d3d0 2023-12-17T06:32:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.466+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc814005260 con 0x7fc82c15d3d0 2023-12-17T06:32:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.470+0000 7fc8227fc640 1 -- 172.21.15.28:0/1010211278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fc824014980 con 0x7fc82c15d3d0 2023-12-17T06:32:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.470+0000 7fc8227fc640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc8100664a0 0x7fc810068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.470+0000 7fc8227fc640 1 -- 172.21.15.28:0/1010211278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc824084bb0 con 0x7fc82c15d3d0 2023-12-17T06:32:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.470+0000 7fc831966640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc8100664a0 0x7fc810068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.470+0000 7fc831966640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc8100664a0 0x7fc810068960 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc8280ae000 tx=0x7fc8280b5040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.478+0000 7fc8227fc640 1 -- 172.21.15.28:0/1010211278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fc824052550 con 0x7fc82c15d3d0 2023-12-17T06:32:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc814000bd0 con 0x7fc8100664a0 2023-12-17T06:32:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc8227fc640 1 -- 172.21.15.28:0/1010211278 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc814000bd0 con 0x7fc8100664a0 2023-12-17T06:32:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 3 to deep-scrub 2023-12-17T06:32:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc8100664a0 msgr2=0x7fc810068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc8100664a0 0x7fc810068960 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc8280ae000 tx=0x7fc8280b5040 comp rx=0 tx=0).stop 2023-12-17T06:32:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 msgr2=0x7fc82c0ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ceb90 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fc82400efc0 tx=0x7fc82400c490 comp rx=0 tx=0).stop 2023-12-17T06:32:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 shutdown_connections 2023-12-17T06:32:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fc8100664a0 0x7fc810068960 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc82c15d3d0 0x7fc82c0ceb90 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fc82c0af0a0 0x7fc82c1547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 --2- 172.21.15.28:0/1010211278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc82c06a530 0x7fc82c0cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 >> 172.21.15.28:0/1010211278 conn(0x7fc82c06d530 msgr2=0x7fc82c06d910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 shutdown_connections 2023-12-17T06:32:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.634+0000 7fc833bca640 1 -- 172.21.15.28:0/1010211278 wait complete. 2023-12-17T06:32:23.657 INFO:tasks.ceph:Scrubbing osd.4 2023-12-17T06:32:23.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-17T06:32:23.764 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5022916640 1 Processor -- start 2023-12-17T06:32:23.765 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5022916640 1 -- start start 2023-12-17T06:32:23.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5022916640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0cda00 0x7f501c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5022916640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f501c0b68f0 0x7f501c0b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5022916640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c0ce3a0 con 0x7f501c160990 2023-12-17T06:32:23.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5022916640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501c0d3b40 con 0x7f501c0b68f0 2023-12-17T06:32:23.766 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5022916640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c0b6460 con 0x7f501c0cda00 2023-12-17T06:32:23.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f501b7fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0cda00 0x7f501c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5020eb3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f501c0b68f0 0x7f501c0b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f501b7fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0cda00 0x7f501c0cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37348/0 (socket says 172.21.15.28:37348) 2023-12-17T06:32:23.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f501b7fe640 1 -- 172.21.15.28:0/450610193 learned_addr learned my addr 172.21.15.28:0/450610193 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:23.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f5020eb3640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f501c0b68f0 0x7f501c0b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.28:34468/0 (socket says 172.21.15.28:34468) 2023-12-17T06:32:23.767 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f501bfff640 1 --1- 172.21.15.28:0/450610193 >> v1:172.21.15.122:6789/0 conn(0x7f501c160990 0x7f501c0d2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.28:48656/0 (socket says 172.21.15.28:48656) 2023-12-17T06:32:23.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f501b7fe640 1 -- 172.21.15.28:0/450610193 >> v2:172.21.15.122:3300/0 conn(0x7f501c0b68f0 msgr2=0x7f501c0b6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:23.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f501b7fe640 1 --2- 172.21.15.28:0/450610193 >> v2:172.21.15.122:3300/0 conn(0x7f501c0b68f0 0x7f501c0b6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f501b7fe640 1 -- 172.21.15.28:0/450610193 >> v1:172.21.15.122:6789/0 conn(0x7f501c160990 legacy=0x7f501c0d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.758+0000 7f501b7fe640 1 -- 172.21.15.28:0/450610193 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c0d32a0 con 0x7f501c0cda00 2023-12-17T06:32:23.768 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f501b7fe640 1 --2- 172.21.15.28:0/450610193 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0cda00 0x7f501c0cde60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5010009870 tx=0x7f501001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69fed55b8ad933eb server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.769 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f501affd640 1 -- 172.21.15.28:0/450610193 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50100ab600 con 0x7f501c0cda00 2023-12-17T06:32:23.769 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f501affd640 1 -- 172.21.15.28:0/450610193 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50100a7050 con 0x7f501c0cda00 2023-12-17T06:32:23.769 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f501affd640 1 -- 172.21.15.28:0/450610193 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50100ab980 con 0x7f501c0cda00 2023-12-17T06:32:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f5022916640 1 -- 172.21.15.28:0/450610193 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0cda00 msgr2=0x7f501c0cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f5022916640 1 --2- 172.21.15.28:0/450610193 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0cda00 0x7f501c0cde60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5010009870 tx=0x7f501001c540 comp rx=0 tx=0).stop 2023-12-17T06:32:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f5022916640 1 -- 172.21.15.28:0/450610193 shutdown_connections 2023-12-17T06:32:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f5022916640 1 --2- 172.21.15.28:0/450610193 >> v2:172.21.15.122:3300/0 conn(0x7f501c0b68f0 0x7f501c0b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f5022916640 1 --2- 172.21.15.28:0/450610193 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0cda00 0x7f501c0cde60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f5022916640 1 -- 172.21.15.28:0/450610193 >> 172.21.15.28:0/450610193 conn(0x7f501c072060 msgr2=0x7f501c072470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f5022916640 1 -- 172.21.15.28:0/450610193 shutdown_connections 2023-12-17T06:32:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.762+0000 7f5022916640 1 -- 172.21.15.28:0/450610193 wait complete. 2023-12-17T06:32:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 Processor -- start 2023-12-17T06:32:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 -- start start 2023-12-17T06:32:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0b68f0 0x7f501c1f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c0cda00 0x7f501c15cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f501c160990 0x7f501c15d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c159730 con 0x7f501c0cda00 2023-12-17T06:32:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c0cb830 con 0x7f501c160990 2023-12-17T06:32:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c1f6fa0 con 0x7f501c0b68f0 2023-12-17T06:32:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c0cda00 0x7f501c15cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c0cda00 0x7f501c15cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36802/0 (socket says 172.21.15.28:36802) 2023-12-17T06:32:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 -- 172.21.15.28:0/3291016928 learned_addr learned my addr 172.21.15.28:0/3291016928 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5020eb3640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f501c160990 0x7f501c15d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501bfff640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0b68f0 0x7f501c1f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 -- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0b68f0 msgr2=0x7f501c1f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f501c0b68f0 0x7f501c1f6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 -- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f501c160990 msgr2=0x7f501c15d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f501c160990 0x7f501c15d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 -- 172.21.15.28:0/3291016928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50100094d0 con 0x7f501c0cda00 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f501b7fe640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c0cda00 0x7f501c15cfc0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f501001a170 tx=0x7f50100192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50100b4420 con 0x7f501c0cda00 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50100b4580 con 0x7f501c0cda00 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501c159ed0 con 0x7f501c0cda00 2023-12-17T06:32:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50100b48f0 con 0x7f501c0cda00 2023-12-17T06:32:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501c159bf0 con 0x7f501c0cda00 2023-12-17T06:32:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.766+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f501c0d2f10 con 0x7f501c0cda00 2023-12-17T06:32:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f50100b2070 con 0x7f501c0cda00 2023-12-17T06:32:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f5018ff9640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5008066470 0x7f5008068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f501011d4c0 con 0x7f501c0cda00 2023-12-17T06:32:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f5018ff9640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f500806c2d0 0x7f500806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 --> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f500806eda0 con 0x7f500806c2d0 2023-12-17T06:32:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f501011d830 con 0x7f501c0cda00 2023-12-17T06:32:23.778 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f501bfff640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5008066470 0x7f5008068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f5020eb3640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f500806c2d0 0x7f500806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f501bfff640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5008066470 0x7f5008068930 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f500c000f70 tx=0x7f500c007e50 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.770+0000 7f5020eb3640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f500806c2d0 0x7f500806e6f0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.774+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 <== osd.4 v2:172.21.15.122:6800/1281108383 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f500806eda0 con 0x7f500806c2d0 2023-12-17T06:32:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 --> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f501c0cb590 con 0x7f500806c2d0 2023-12-17T06:32:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5018ff9640 1 -- 172.21.15.28:0/3291016928 <== osd.4 v2:172.21.15.122:6800/1281108383 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f501c0cb590 con 0x7f500806c2d0 2023-12-17T06:32:23.793 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-17T06:32:23.793 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-17T06:32:23.793 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-17T06:32:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f500806c2d0 msgr2=0x7f500806e6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6800/1281108383,v1:172.21.15.122:6801/1281108383] conn(0x7f500806c2d0 0x7f500806e6f0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5008066470 msgr2=0x7f5008068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f5008066470 0x7f5008068930 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f500c000f70 tx=0x7f500c007e50 comp rx=0 tx=0).stop 2023-12-17T06:32:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c0cda00 msgr2=0x7f501c15cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 --2- 172.21.15.28:0/3291016928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c0cda00 0x7f501c15cfc0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f501001a170 tx=0x7f50100192e0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5020eb3640 1 -- 172.21.15.28:0/3291016928 reap_dead start 2023-12-17T06:32:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 shutdown_connections 2023-12-17T06:32:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 >> 172.21.15.28:0/3291016928 conn(0x7f501c072060 msgr2=0x7f501c0b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 shutdown_connections 2023-12-17T06:32:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.786+0000 7f5022916640 1 -- 172.21.15.28:0/3291016928 wait complete. 2023-12-17T06:32:23.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-17T06:32:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7426e16640 1 Processor -- start 2023-12-17T06:32:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7426e16640 1 -- start start 2023-12-17T06:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7426e16640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74200b36f0 0x7f742015d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7426e16640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f742006e950 0x7f742006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7426e16640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742015df50 con 0x7f7420071f50 2023-12-17T06:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7426e16640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7420160d10 con 0x7f74200b36f0 2023-12-17T06:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7426e16640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742015e230 con 0x7f742006e950 2023-12-17T06:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7417fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7420071f50 0x7f7420161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46152/0 (socket says 172.21.15.28:46152) 2023-12-17T06:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7424bb2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74200b36f0 0x7f742015d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7417fff640 1 -- 172.21.15.28:0/546290719 learned_addr learned my addr 172.21.15.28:0/546290719 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f74253b3640 1 --2- 172.21.15.28:0/546290719 >> v2:172.21.15.122:3300/0 conn(0x7f742006e950 0x7f742006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7424bb2640 1 -- 172.21.15.28:0/546290719 >> v2:172.21.15.122:3300/0 conn(0x7f742006e950 msgr2=0x7f742006ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7424bb2640 1 --2- 172.21.15.28:0/546290719 >> v2:172.21.15.122:3300/0 conn(0x7f742006e950 0x7f742006ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7424bb2640 1 -- 172.21.15.28:0/546290719 >> v1:172.21.15.28:6789/0 conn(0x7f7420071f50 legacy=0x7f7420161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f74177fe640 1 -- 172.21.15.28:0/546290719 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968435663 0 0) 0x7f742015df50 con 0x7f7420071f50 2023-12-17T06:32:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.910+0000 7f7424bb2640 1 -- 172.21.15.28:0/546290719 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742015e510 con 0x7f74200b36f0 2023-12-17T06:32:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7424bb2640 1 --2- 172.21.15.28:0/546290719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f742015d720 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f742015b2f0 tx=0x7f741c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811045cda26ea5d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74177fe640 1 -- 172.21.15.28:0/546290719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0ab600 con 0x7f74200b36f0 2023-12-17T06:32:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74177fe640 1 -- 172.21.15.28:0/546290719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c0a7050 con 0x7f74200b36f0 2023-12-17T06:32:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74177fe640 1 -- 172.21.15.28:0/546290719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0ab980 con 0x7f74200b36f0 2023-12-17T06:32:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- 172.21.15.28:0/546290719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 msgr2=0x7f742015d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 --2- 172.21.15.28:0/546290719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f742015d720 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f742015b2f0 tx=0x7f741c01c460 comp rx=0 tx=0).stop 2023-12-17T06:32:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- 172.21.15.28:0/546290719 shutdown_connections 2023-12-17T06:32:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 --2- 172.21.15.28:0/546290719 >> v2:172.21.15.122:3300/0 conn(0x7f742006e950 0x7f742006ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 --2- 172.21.15.28:0/546290719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f742015d720 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- 172.21.15.28:0/546290719 >> 172.21.15.28:0/546290719 conn(0x7f74200cfe90 msgr2=0x7f74200d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- 172.21.15.28:0/546290719 shutdown_connections 2023-12-17T06:32:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- 172.21.15.28:0/546290719 wait complete. 2023-12-17T06:32:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 Processor -- start 2023-12-17T06:32:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- start start 2023-12-17T06:32:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f742006e950 0x7f742014aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7420071f50 0x7f7420158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f7420158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742015c790 con 0x7f74200b36f0 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742015fb90 con 0x7f7420071f50 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7426e16640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742015be60 con 0x7f742006e950 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f7420158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f7420158b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36832/0 (socket says 172.21.15.28:36832) 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 -- 172.21.15.28:0/351493366 learned_addr learned my addr 172.21.15.28:0/351493366 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7424bb2640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f742006e950 0x7f742014aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7417fff640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7420071f50 0x7f7420158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 -- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f742006e950 msgr2=0x7f742014aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f742006e950 0x7f742014aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 -- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7420071f50 msgr2=0x7f7420158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7420071f50 0x7f7420158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 -- 172.21.15.28:0/351493366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c009420 con 0x7f74200b36f0 2023-12-17T06:32:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f7417fff640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7420071f50 0x7f7420158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-17T06:32:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74253b3640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f7420158b40 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f7418007e40 tx=0x7f741800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74157fa640 1 -- 172.21.15.28:0/351493366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7418014620 con 0x7f74200b36f0 2023-12-17T06:32:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.914+0000 7f74157fa640 1 -- 172.21.15.28:0/351493366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741800ee90 con 0x7f74200b36f0 2023-12-17T06:32:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742015c140 con 0x7f74200b36f0 2023-12-17T06:32:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f74157fa640 1 -- 172.21.15.28:0/351493366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74180149e0 con 0x7f74200b36f0 2023-12-17T06:32:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742014b3f0 con 0x7f74200b36f0 2023-12-17T06:32:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74200674d0 con 0x7f74200b36f0 2023-12-17T06:32:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f74157fa640 1 -- 172.21.15.28:0/351493366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f7418014b40 con 0x7f74200b36f0 2023-12-17T06:32:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f74157fa640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f740c066470 0x7f740c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f74157fa640 1 -- 172.21.15.28:0/351493366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7418084530 con 0x7f74200b36f0 2023-12-17T06:32:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f7424bb2640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f740c066470 0x7f740c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.918+0000 7f7424bb2640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f740c066470 0x7f740c068930 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f741c0ad000 tx=0x7f741c0a6000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:23.922+0000 7f74157fa640 1 -- 172.21.15.28:0/351493366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f7418051c20 con 0x7f74200b36f0 2023-12-17T06:32:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f742015bfd0 con 0x7f740c066470 2023-12-17T06:32:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f74157fa640 1 -- 172.21.15.28:0/351493366 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f742015bfd0 con 0x7f740c066470 2023-12-17T06:32:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 4 to deep-scrub 2023-12-17T06:32:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f740c066470 msgr2=0x7f740c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f740c066470 0x7f740c068930 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f741c0ad000 tx=0x7f741c0a6000 comp rx=0 tx=0).stop 2023-12-17T06:32:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 msgr2=0x7f7420158b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f7420158b40 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f7418007e40 tx=0x7f741800e4b0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 shutdown_connections 2023-12-17T06:32:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f740c066470 0x7f740c068930 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74200b36f0 0x7f7420158b40 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7420071f50 0x7f7420158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 --2- 172.21.15.28:0/351493366 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f742006e950 0x7f742014aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 >> 172.21.15.28:0/351493366 conn(0x7f74200cfe90 msgr2=0x7f74200ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 shutdown_connections 2023-12-17T06:32:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.074+0000 7f7426e16640 1 -- 172.21.15.28:0/351493366 wait complete. 2023-12-17T06:32:24.103 INFO:tasks.ceph:Scrubbing osd.5 2023-12-17T06:32:24.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-17T06:32:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.210+0000 7fb565059640 1 Processor -- start 2023-12-17T06:32:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.210+0000 7fb565059640 1 -- start start 2023-12-17T06:32:24.219 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb56015a030 0x7fb56015e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5600cb9e0 0x7fb56006b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56015a410 con 0x7fb56006ba80 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5600ce220 con 0x7fb5600cb9e0 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56006bf10 con 0x7fb56015a030 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5600cb9e0 0x7fb56006b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55f59e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb56006ba80 0x7fb5600b3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46164/0 (socket says 172.21.15.28:46164) 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55f59e640 1 -- 172.21.15.28:0/1692606795 learned_addr learned my addr 172.21.15.28:0/1692606795 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/1692606795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb56015a030 0x7fb56015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55e59c640 1 -- 172.21.15.28:0/1692606795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb56015a030 msgr2=0x7fb56015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55e59c640 1 --2- 172.21.15.28:0/1692606795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb56015a030 0x7fb56015e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55e59c640 1 -- 172.21.15.28:0/1692606795 >> v1:172.21.15.28:6789/0 conn(0x7fb56006ba80 legacy=0x7fb5600b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55e59c640 1 -- 172.21.15.28:0/1692606795 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5600cc150 con 0x7fb5600cb9e0 2023-12-17T06:32:24.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55dd9b640 1 -- 172.21.15.28:0/1692606795 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085129703 0 0) 0x7fb56015a410 con 0x7fb56006ba80 2023-12-17T06:32:24.221 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55e59c640 1 --2- 172.21.15.28:0/1692606795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5600cb9e0 0x7fb56006b540 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fb5500097c0 tx=0x7fb55001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3cddd03b1d401e server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55dd9b640 1 -- 172.21.15.28:0/1692606795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500ab600 con 0x7fb5600cb9e0 2023-12-17T06:32:24.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55dd9b640 1 -- 172.21.15.28:0/1692606795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5500a7050 con 0x7fb5600cb9e0 2023-12-17T06:32:24.222 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb55dd9b640 1 -- 172.21.15.28:0/1692606795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500ab980 con 0x7fb5600cb9e0 2023-12-17T06:32:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 -- 172.21.15.28:0/1692606795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5600cb9e0 msgr2=0x7fb56006b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 --2- 172.21.15.28:0/1692606795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5600cb9e0 0x7fb56006b540 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fb5500097c0 tx=0x7fb55001c450 comp rx=0 tx=0).stop 2023-12-17T06:32:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 -- 172.21.15.28:0/1692606795 shutdown_connections 2023-12-17T06:32:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 --2- 172.21.15.28:0/1692606795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5600cb9e0 0x7fb56006b540 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 --2- 172.21.15.28:0/1692606795 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb56015a030 0x7fb56015e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 -- 172.21.15.28:0/1692606795 >> 172.21.15.28:0/1692606795 conn(0x7fb560070cd0 msgr2=0x7fb5600710e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 -- 172.21.15.28:0/1692606795 shutdown_connections 2023-12-17T06:32:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.214+0000 7fb565059640 1 -- 172.21.15.28:0/1692606795 wait complete. 2023-12-17T06:32:24.225 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 Processor -- start 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 -- start start 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56006ba80 0x7fb5601f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb5600cb9e0 0x7fb560158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb56015a030 0x7fb56006dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5601f6900 con 0x7fb56006ba80 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb560148ec0 con 0x7fb56015a030 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5601482b0 con 0x7fb5600cb9e0 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56006ba80 0x7fb5601f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.226 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56006ba80 0x7fb5601f63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36860/0 (socket says 172.21.15.28:36860) 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 -- 172.21.15.28:0/2425420303 learned_addr learned my addr 172.21.15.28:0/2425420303 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55e59c640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb5600cb9e0 0x7fb560158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55f59e640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb56015a030 0x7fb56006dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 -- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb5600cb9e0 msgr2=0x7fb560158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fb5600cb9e0 0x7fb560158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 -- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb56015a030 msgr2=0x7fb56006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb56015a030 0x7fb56006dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 -- 172.21.15.28:0/2425420303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550009420 con 0x7fb56006ba80 2023-12-17T06:32:24.227 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56006ba80 0x7fb5601f63c0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fb55400e870 tx=0x7fb55400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554009510 con 0x7fb56006ba80 2023-12-17T06:32:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55400eec0 con 0x7fb56006ba80 2023-12-17T06:32:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb560148590 con 0x7fb56006ba80 2023-12-17T06:32:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55400cb40 con 0x7fb56006ba80 2023-12-17T06:32:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb560148b50 con 0x7fb56006ba80 2023-12-17T06:32:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5600cbe40 con 0x7fb56006ba80 2023-12-17T06:32:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.218+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fb55400cca0 con 0x7fb56006ba80 2023-12-17T06:32:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb5477fe640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb54c066470 0x7fb54c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb554083cb0 con 0x7fb56006ba80 2023-12-17T06:32:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb5477fe640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fb54c06c2d0 0x7fb54c06e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 --> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb54c06eda0 con 0x7fb54c06c2d0 2023-12-17T06:32:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb554086370 con 0x7fb56006ba80 2023-12-17T06:32:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb55e59c640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb54c066470 0x7fb54c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb55f59e640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fb54c06c2d0 0x7fb54c06e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb55f59e640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fb54c06c2d0 0x7fb54c06e6f0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb55e59c640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb54c066470 0x7fb54c068930 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fb5500ad000 tx=0x7fb5500a6000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.222+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 <== osd.5 v2:172.21.15.122:6816/1068560627 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb54c06eda0 con 0x7fb54c06c2d0 2023-12-17T06:32:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.234+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 --> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb5600674d0 con 0x7fb54c06c2d0 2023-12-17T06:32:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.234+0000 7fb5477fe640 1 -- 172.21.15.28:0/2425420303 <== osd.5 v2:172.21.15.122:6816/1068560627 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb5600674d0 con 0x7fb54c06c2d0 2023-12-17T06:32:24.243 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-17T06:32:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-17T06:32:24.243 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-17T06:32:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fb54c06c2d0 msgr2=0x7fb54c06e6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6816/1068560627,v1:172.21.15.122:6817/1068560627] conn(0x7fb54c06c2d0 0x7fb54c06e6f0 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb54c066470 msgr2=0x7fb54c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.244 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fb54c066470 0x7fb54c068930 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fb5500ad000 tx=0x7fb5500a6000 comp rx=0 tx=0).stop 2023-12-17T06:32:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56006ba80 msgr2=0x7fb5601f63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 --2- 172.21.15.28:0/2425420303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb56006ba80 0x7fb5601f63c0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fb55400e870 tx=0x7fb55400ec70 comp rx=0 tx=0).stop 2023-12-17T06:32:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb55f59e640 1 -- 172.21.15.28:0/2425420303 reap_dead start 2023-12-17T06:32:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 shutdown_connections 2023-12-17T06:32:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 >> 172.21.15.28:0/2425420303 conn(0x7fb560070cd0 msgr2=0x7fb5600710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 shutdown_connections 2023-12-17T06:32:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.238+0000 7fb565059640 1 -- 172.21.15.28:0/2425420303 wait complete. 2023-12-17T06:32:24.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-17T06:32:24.363 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f0831f4c640 1 Processor -- start 2023-12-17T06:32:24.364 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f0831f4c640 1 -- start start 2023-12-17T06:32:24.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f0831f4c640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f082c15d240 0x7f082c0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f0831f4c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 0x7f082c0ad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f0831f4c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c0adca0 con 0x7f082c06cda0 2023-12-17T06:32:24.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f0831f4c640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082c0adf80 con 0x7f082c15d240 2023-12-17T06:32:24.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f0831f4c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0ae260 con 0x7f082c06bc50 2023-12-17T06:32:24.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082affd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f082c06cda0 0x7f082c06b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46180/0 (socket says 172.21.15.28:46180) 2023-12-17T06:32:24.367 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082affd640 1 -- 172.21.15.28:0/2910904766 learned_addr learned my addr 172.21.15.28:0/2910904766 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082b7fe640 1 --2- 172.21.15.28:0/2910904766 >> v2:172.21.15.122:3300/0 conn(0x7f082c15d240 0x7f082c0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082bfff640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 0x7f082c0ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813146212 0 0) 0x7f082c0adca0 con 0x7f082c06cda0 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0814003530 con 0x7f082c06cda0 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1112885291 0 0) 0x7f0814003530 con 0x7f082c06cda0 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 msgr2=0x7f082c0ad630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 0x7f082c0ad630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 >> v2:172.21.15.122:3300/0 conn(0x7f082c15d240 msgr2=0x7f082c0aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 --2- 172.21.15.28:0/2910904766 >> v2:172.21.15.122:3300/0 conn(0x7f082c15d240 0x7f082c0aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c0ae540 con 0x7f082c06cda0 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f0820001680 con 0x7f082c06cda0 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 >> v1:172.21.15.28:6789/0 conn(0x7f082c06cda0 legacy=0x7f082c06b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0814003980 0x7f0814003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140047d0 0x7f0814004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 0x7f0814005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0adf80 con 0x7f0814003980 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0ae260 con 0x7f08140047d0 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081400b9c0 con 0x7f082c06bc50 2023-12-17T06:32:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082b7fe640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0814003980 0x7f0814003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082affd640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140047d0 0x7f0814004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082bfff640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 0x7f0814005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082b7fe640 1 -- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 msgr2=0x7f0814005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082b7fe640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 0x7f0814005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082b7fe640 1 -- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140047d0 msgr2=0x7f0814004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082b7fe640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08140047d0 0x7f0814004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082b7fe640 1 -- 172.21.15.28:0/2910904766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814003530 con 0x7f0814003980 2023-12-17T06:32:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.358+0000 7f082bfff640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f082c06bc50 0x7f0814005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-17T06:32:24.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f082b7fe640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0814003980 0x7f0814003da0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f081c009fc0 tx=0x7f081c01cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f57de68e66b2a6 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.373 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c0a5a90 con 0x7f0814003980 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081c00f040 con 0x7f0814003980 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f082a7fc640 1 -- 172.21.15.28:0/2910904766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c0a5e30 con 0x7f0814003980 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f0831f4c640 1 -- 172.21.15.28:0/2910904766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0814003980 msgr2=0x7f0814003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f0831f4c640 1 --2- 172.21.15.28:0/2910904766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0814003980 0x7f0814003da0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f081c009fc0 tx=0x7f081c01cc10 comp rx=0 tx=0).stop 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f082bfff640 1 -- 172.21.15.28:0/2910904766 reap_dead start 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f0831f4c640 1 -- 172.21.15.28:0/2910904766 shutdown_connections 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f0831f4c640 1 -- 172.21.15.28:0/2910904766 >> 172.21.15.28:0/2910904766 conn(0x7f082c070d40 msgr2=0x7f082c071150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f0831f4c640 1 -- 172.21.15.28:0/2910904766 shutdown_connections 2023-12-17T06:32:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f0831f4c640 1 -- 172.21.15.28:0/2910904766 wait complete. 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f0831f4c640 1 Processor -- start 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.362+0000 7f0831f4c640 1 -- start start 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0814003980 0x7f082c152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c157c20 0x7f082c160df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c161330 0x7f082c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c1617b0 con 0x7f082c161330 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c160280 con 0x7f082c157c20 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c160740 con 0x7f0814003980 2023-12-17T06:32:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c161330 0x7f082c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c161330 0x7f082c1585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36882/0 (socket says 172.21.15.28:36882) 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 -- 172.21.15.28:0/603076253 learned_addr learned my addr 172.21.15.28:0/603076253 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082b7fe640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0814003980 0x7f082c152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082affd640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c157c20 0x7f082c160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 -- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0814003980 msgr2=0x7f082c152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0814003980 0x7f082c152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 -- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c157c20 msgr2=0x7f082c160df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c157c20 0x7f082c160df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 -- 172.21.15.28:0/603076253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c153220 con 0x7f082c161330 2023-12-17T06:32:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082bfff640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c161330 0x7f082c1585b0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f082c06d230 tx=0x7f0824010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0813fff640 1 -- 172.21.15.28:0/603076253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082400e8d0 con 0x7f082c161330 2023-12-17T06:32:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0813fff640 1 -- 172.21.15.28:0/603076253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082400ea30 con 0x7f082c161330 2023-12-17T06:32:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082c15f670 con 0x7f082c161330 2023-12-17T06:32:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0813fff640 1 -- 172.21.15.28:0/603076253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082400edc0 con 0x7f082c161330 2023-12-17T06:32:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082c15fc30 con 0x7f082c161330 2023-12-17T06:32:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082c0674d0 con 0x7f082c161330 2023-12-17T06:32:24.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0813fff640 1 -- 172.21.15.28:0/603076253 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f0824021460 con 0x7f082c161330 2023-12-17T06:32:24.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0813fff640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f08180664a0 0x7f0818068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.378 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f0813fff640 1 -- 172.21.15.28:0/603076253 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f08240839f0 con 0x7f082c161330 2023-12-17T06:32:24.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.366+0000 7f082b7fe640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f08180664a0 0x7f0818068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.370+0000 7f082b7fe640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f08180664a0 0x7f0818068960 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f081c0af000 tx=0x7f081c0a7040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.370+0000 7f0813fff640 1 -- 172.21.15.28:0/603076253 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f08240510e0 con 0x7f082c161330 2023-12-17T06:32:24.528 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f082c06e5f0 con 0x7f08180664a0 2023-12-17T06:32:24.528 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0813fff640 1 -- 172.21.15.28:0/603076253 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f082c06e5f0 con 0x7f08180664a0 2023-12-17T06:32:24.528 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 5 to deep-scrub 2023-12-17T06:32:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f08180664a0 msgr2=0x7f0818068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f08180664a0 0x7f0818068960 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f081c0af000 tx=0x7f081c0a7040 comp rx=0 tx=0).stop 2023-12-17T06:32:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c161330 msgr2=0x7f082c1585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c161330 0x7f082c1585b0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f082c06d230 tx=0x7f0824010c20 comp rx=0 tx=0).stop 2023-12-17T06:32:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 shutdown_connections 2023-12-17T06:32:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f08180664a0 0x7f0818068960 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f082c161330 0x7f082c1585b0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c157c20 0x7f082c160df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 --2- 172.21.15.28:0/603076253 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0814003980 0x7f082c152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 >> 172.21.15.28:0/603076253 conn(0x7f082c070d40 msgr2=0x7f082c153420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 shutdown_connections 2023-12-17T06:32:24.532 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.522+0000 7f0831f4c640 1 -- 172.21.15.28:0/603076253 wait complete. 2023-12-17T06:32:24.546 INFO:tasks.ceph:Scrubbing osd.6 2023-12-17T06:32:24.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-17T06:32:24.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2913abc640 1 Processor -- start 2023-12-17T06:32:24.661 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2913abc640 1 -- start start 2023-12-17T06:32:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2913abc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f290c146b60 0x7f290c06a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2913abc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 0x7f290c0b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2913abc640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290c0b5090 con 0x7f290c0b4000 2023-12-17T06:32:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2913abc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290c0ad980 con 0x7f290c146b60 2023-12-17T06:32:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2913abc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0b47f0 con 0x7f290c06aa70 2023-12-17T06:32:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2912059640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f290c0b4000 0x7f290c071540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46182/0 (socket says 172.21.15.28:46182) 2023-12-17T06:32:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2912059640 1 -- 172.21.15.28:0/1680852372 learned_addr learned my addr 172.21.15.28:0/1680852372 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2911057640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 0x7f290c0b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2911858640 1 --2- 172.21.15.28:0/1680852372 >> v2:172.21.15.122:3300/0 conn(0x7f290c146b60 0x7f290c06a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008508684 0 0) 0x7f290c0b5090 con 0x7f290c0b4000 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.654+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f8003530 con 0x7f290c0b4000 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3124594443 0 0) 0x7f28f8003530 con 0x7f290c0b4000 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 msgr2=0x7f290c0b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 0x7f290c0b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 >> v2:172.21.15.122:3300/0 conn(0x7f290c146b60 msgr2=0x7f290c06a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 --2- 172.21.15.28:0/1680852372 >> v2:172.21.15.122:3300/0 conn(0x7f290c146b60 0x7f290c06a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c06aed0 con 0x7f290c0b4000 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f2904001a00 con 0x7f290c0b4000 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 >> v1:172.21.15.28:6789/0 conn(0x7f290c0b4000 legacy=0x7f290c071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f8003980 0x7f28f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 0x7f28f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f28f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0ad980 con 0x7f28f8004d10 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0b47f0 con 0x7f28f8003980 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f800b990 con 0x7f290c06aa70 2023-12-17T06:32:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2912059640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f28f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2911858640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f8003980 0x7f28f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2911057640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 0x7f28f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2912059640 1 -- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 msgr2=0x7f28f80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2912059640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 0x7f28f80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2912059640 1 -- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f8003980 msgr2=0x7f28f8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2912059640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f8003980 0x7f28f8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2912059640 1 -- 172.21.15.28:0/1680852372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8003530 con 0x7f28f8004d10 2023-12-17T06:32:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2911057640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c06aa70 0x7f28f80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-17T06:32:24.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2912059640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f28f8005190 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f290400a830 tx=0x7f290401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb0f6902f56b985 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.670 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29040ad630 con 0x7f28f8004d10 2023-12-17T06:32:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29040a8070 con 0x7f28f8004d10 2023-12-17T06:32:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2910856640 1 -- 172.21.15.28:0/1680852372 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29040ad9d0 con 0x7f28f8004d10 2023-12-17T06:32:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2913abc640 1 -- 172.21.15.28:0/1680852372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 msgr2=0x7f28f8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2913abc640 1 --2- 172.21.15.28:0/1680852372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f28f8005190 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f290400a830 tx=0x7f290401d340 comp rx=0 tx=0).stop 2023-12-17T06:32:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2912059640 1 -- 172.21.15.28:0/1680852372 reap_dead start 2023-12-17T06:32:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2913abc640 1 -- 172.21.15.28:0/1680852372 shutdown_connections 2023-12-17T06:32:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2913abc640 1 -- 172.21.15.28:0/1680852372 >> 172.21.15.28:0/1680852372 conn(0x7f290c0b1db0 msgr2=0x7f290c0b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2913abc640 1 -- 172.21.15.28:0/1680852372 shutdown_connections 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.658+0000 7f2913abc640 1 -- 172.21.15.28:0/1680852372 wait complete. 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 Processor -- start 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 -- start start 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f290c0b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c0c5020 0x7f290c0c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15fca0 0x7f290c15c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0c63a0 con 0x7f28f8004d10 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0c5790 con 0x7f290c15fca0 2023-12-17T06:32:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290c0c5a70 con 0x7f290c0c5020 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f290c0b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f290c0b10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36898/0 (socket says 172.21.15.28:36898) 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 -- 172.21.15.28:0/3331375545 learned_addr learned my addr 172.21.15.28:0/3331375545 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911057640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c0c5020 0x7f290c0c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2912059640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15fca0 0x7f290c15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 -- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c0c5020 msgr2=0x7f290c0c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f290c0c5020 0x7f290c0c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 -- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15fca0 msgr2=0x7f290c15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15fca0 0x7f290c15c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 -- 172.21.15.28:0/3331375545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c0c5d50 con 0x7f28f8004d10 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2912059640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f290c15fca0 0x7f290c15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2911858640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f290c0b10e0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f29080119d0 tx=0x7f2908011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290800fb50 con 0x7f28f8004d10 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290800fcb0 con 0x7f28f8004d10 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908014440 con 0x7f28f8004d10 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290c0c6030 con 0x7f28f8004d10 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290c0c6d70 con 0x7f28f8004d10 2023-12-17T06:32:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.662+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f290c0ad5a0 con 0x7f28f8004d10 2023-12-17T06:32:24.675 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f290801d400 con 0x7f28f8004d10 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f29027fc640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28f0066470 0x7f28f0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2908084a20 con 0x7f28f8004d10 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f29027fc640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f28f006c2d0 0x7f28f006e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 --> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28f006eda0 con 0x7f28f006c2d0 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2908084e70 con 0x7f28f8004d10 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f2911057640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28f0066470 0x7f28f0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f2912059640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f28f006c2d0 0x7f28f006e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f2911057640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28f0066470 0x7f28f0068930 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f290c0b1950 tx=0x7f28fc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f2912059640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f28f006c2d0 0x7f28f006e6f0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.666+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 <== osd.6 v2:172.21.15.122:6824/1879423542 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f28f006eda0 con 0x7f28f006c2d0 2023-12-17T06:32:24.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.682+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 --> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f290c0b4db0 con 0x7f28f006c2d0 2023-12-17T06:32:24.690 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.682+0000 7f29027fc640 1 -- 172.21.15.28:0/3331375545 <== osd.6 v2:172.21.15.122:6824/1879423542 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f29040017c0 con 0x7f28f006c2d0 2023-12-17T06:32:24.691 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-17T06:32:24.691 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-17T06:32:24.691 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-17T06:32:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f28f006c2d0 msgr2=0x7f28f006e6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6824/1879423542,v1:172.21.15.122:6825/1879423542] conn(0x7f28f006c2d0 0x7f28f006e6f0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28f0066470 msgr2=0x7f28f0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f28f0066470 0x7f28f0068930 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f290c0b1950 tx=0x7f28fc008040 comp rx=0 tx=0).stop 2023-12-17T06:32:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 msgr2=0x7f290c0b10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 --2- 172.21.15.28:0/3331375545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f8004d10 0x7f290c0b10e0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f29080119d0 tx=0x7f2908011e90 comp rx=0 tx=0).stop 2023-12-17T06:32:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2912059640 1 -- 172.21.15.28:0/3331375545 reap_dead start 2023-12-17T06:32:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 shutdown_connections 2023-12-17T06:32:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 >> 172.21.15.28:0/3331375545 conn(0x7f290c0b1db0 msgr2=0x7f290c1424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 shutdown_connections 2023-12-17T06:32:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.686+0000 7f2913abc640 1 -- 172.21.15.28:0/3331375545 wait complete. 2023-12-17T06:32:24.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-17T06:32:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031e74a640 1 Processor -- start 2023-12-17T06:32:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031e74a640 1 -- start start 2023-12-17T06:32:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031e74a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f031806ac40 0x7f03180af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031e74a640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03180ad840 con 0x7f03180ae8a0 2023-12-17T06:32:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031e74a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03180adae0 con 0x7f031815d3d0 2023-12-17T06:32:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031e74a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031815d7b0 con 0x7f031806ac40 2023-12-17T06:32:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f030f7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f03180ae8a0 0x7f031806a530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46194/0 (socket says 172.21.15.28:46194) 2023-12-17T06:32:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031cce7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f031806ac40 0x7f03180af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f030f7fe640 1 -- 172.21.15.28:0/1938815318 learned_addr learned my addr 172.21.15.28:0/1938815318 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031cce7640 1 -- 172.21.15.28:0/1938815318 >> v1:172.21.15.122:6789/0 conn(0x7f031815d3d0 legacy=0x7f031806e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031cce7640 1 -- 172.21.15.28:0/1938815318 >> v1:172.21.15.28:6789/0 conn(0x7f03180ae8a0 legacy=0x7f031806a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f031cce7640 1 -- 172.21.15.28:0/1938815318 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180addc0 con 0x7f031806ac40 2023-12-17T06:32:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.810+0000 7f03177fe640 1 -- 172.21.15.28:0/1938815318 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242228712 0 0) 0x7f03180ad840 con 0x7f03180ae8a0 2023-12-17T06:32:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031cce7640 1 --2- 172.21.15.28:0/1938815318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031806ac40 0x7f03180af0a0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f0310009dc0 tx=0x7f031001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8838e2897c602f85 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f03177fe640 1 -- 172.21.15.28:0/1938815318 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03100a5890 con 0x7f031806ac40 2023-12-17T06:32:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f03177fe640 1 -- 172.21.15.28:0/1938815318 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03100a59f0 con 0x7f031806ac40 2023-12-17T06:32:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f03177fe640 1 -- 172.21.15.28:0/1938815318 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03100a5d80 con 0x7f031806ac40 2023-12-17T06:32:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 -- 172.21.15.28:0/1938815318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031806ac40 msgr2=0x7f03180af0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 --2- 172.21.15.28:0/1938815318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031806ac40 0x7f03180af0a0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f0310009dc0 tx=0x7f031001c8c0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 -- 172.21.15.28:0/1938815318 shutdown_connections 2023-12-17T06:32:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 --2- 172.21.15.28:0/1938815318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031806ac40 0x7f03180af0a0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 -- 172.21.15.28:0/1938815318 >> 172.21.15.28:0/1938815318 conn(0x7f031806d530 msgr2=0x7f031806d940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 -- 172.21.15.28:0/1938815318 shutdown_connections 2023-12-17T06:32:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 -- 172.21.15.28:0/1938815318 wait complete. 2023-12-17T06:32:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 Processor -- start 2023-12-17T06:32:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.814+0000 7f031e74a640 1 -- start start 2023-12-17T06:32:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f031806ac40 0x7f0318158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f03180ae8a0 0x7f03181f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031815d3d0 0x7f031814b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031814b090 con 0x7f031815d3d0 2023-12-17T06:32:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03181fa1d0 con 0x7f031806ac40 2023-12-17T06:32:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03181f9c10 con 0x7f03180ae8a0 2023-12-17T06:32:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031815d3d0 0x7f031814b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031815d3d0 0x7f031814b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36916/0 (socket says 172.21.15.28:36916) 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 -- 172.21.15.28:0/3173986383 learned_addr learned my addr 172.21.15.28:0/3173986383 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f0317fff640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f031806ac40 0x7f0318158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f030f7fe640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f03180ae8a0 0x7f03181f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 -- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f03180ae8a0 msgr2=0x7f03181f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f03180ae8a0 0x7f03181f92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 -- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f031806ac40 msgr2=0x7f0318158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f031806ac40 0x7f0318158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 -- 172.21.15.28:0/3173986383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0310009a70 con 0x7f031815d3d0 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031cce7640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031815d3d0 0x7f031814b4b0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f031001cda0 tx=0x7f0310019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f03157fa640 1 -- 172.21.15.28:0/3173986383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03100ad710 con 0x7f031815d3d0 2023-12-17T06:32:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f03157fa640 1 -- 172.21.15.28:0/3173986383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03100ad870 con 0x7f031815d3d0 2023-12-17T06:32:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03181f9ef0 con 0x7f031815d3d0 2023-12-17T06:32:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f03157fa640 1 -- 172.21.15.28:0/3173986383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03100adbe0 con 0x7f031815d3d0 2023-12-17T06:32:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03181f97e0 con 0x7f031815d3d0 2023-12-17T06:32:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031814b1d0 con 0x7f031815d3d0 2023-12-17T06:32:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f03157fa640 1 -- 172.21.15.28:0/3173986383 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f03100add40 con 0x7f031815d3d0 2023-12-17T06:32:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f03157fa640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f02f8066470 0x7f02f8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f03157fa640 1 -- 172.21.15.28:0/3173986383 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f031011ee10 con 0x7f031815d3d0 2023-12-17T06:32:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.818+0000 7f0317fff640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f02f8066470 0x7f02f8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.822+0000 7f0317fff640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f02f8066470 0x7f02f8068930 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f030800b810 tx=0x7f0308008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.822+0000 7f03157fa640 1 -- 172.21.15.28:0/3173986383 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f031814b1d0 con 0x7f031815d3d0 2023-12-17T06:32:24.979 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f03180b6360 con 0x7f02f8066470 2023-12-17T06:32:24.980 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f03157fa640 1 -- 172.21.15.28:0/3173986383 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f03180b6360 con 0x7f02f8066470 2023-12-17T06:32:24.980 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 6 to deep-scrub 2023-12-17T06:32:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f02f8066470 msgr2=0x7f02f8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f02f8066470 0x7f02f8068930 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f030800b810 tx=0x7f0308008040 comp rx=0 tx=0).stop 2023-12-17T06:32:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031815d3d0 msgr2=0x7f031814b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031815d3d0 0x7f031814b4b0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f031001cda0 tx=0x7f0310019660 comp rx=0 tx=0).stop 2023-12-17T06:32:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 shutdown_connections 2023-12-17T06:32:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f02f8066470 0x7f02f8068930 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031815d3d0 0x7f031814b4b0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f03180ae8a0 0x7f03181f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 --2- 172.21.15.28:0/3173986383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f031806ac40 0x7f0318158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:24.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.974+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 >> 172.21.15.28:0/3173986383 conn(0x7f031806d530 msgr2=0x7f031815bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:24.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.978+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 shutdown_connections 2023-12-17T06:32:24.984 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:24.978+0000 7f031e74a640 1 -- 172.21.15.28:0/3173986383 wait complete. 2023-12-17T06:32:25.004 INFO:tasks.ceph:Scrubbing osd.7 2023-12-17T06:32:25.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-17T06:32:25.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee9ad1b640 1 Processor -- start 2023-12-17T06:32:25.121 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee9ad1b640 1 -- start start 2023-12-17T06:32:25.123 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee9ad1b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 0x7fee940ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee9ad1b640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee940ccea0 con 0x7fee940b3ee0 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee9ad1b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee94144ed0 con 0x7fee94144af0 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee9ad1b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee940d0d50 con 0x7fee9406ef40 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee93fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fee940b3ee0 0x7fee9406e830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46210/0 (socket says 172.21.15.28:46210) 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee93fff640 1 -- 172.21.15.28:0/1735781230 learned_addr learned my addr 172.21.15.28:0/1735781230 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee992b8640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 0x7fee940ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586598892 0 0) 0x7fee940ccea0 con 0x7fee940b3ee0 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee84003530 con 0x7fee940b3ee0 2023-12-17T06:32:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2339418022 0 0) 0x7fee84003530 con 0x7fee940b3ee0 2023-12-17T06:32:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 msgr2=0x7fee940ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 0x7fee940ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 >> v1:172.21.15.122:6789/0 conn(0x7fee94144af0 legacy=0x7fee9415f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee940cd720 con 0x7fee940b3ee0 2023-12-17T06:32:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7fee80001680 con 0x7fee940b3ee0 2023-12-17T06:32:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 >> v1:172.21.15.28:6789/0 conn(0x7fee940b3ee0 legacy=0x7fee9406e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 0x7fee840039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee84003f50 0x7fee840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84004d10 0x7fee84005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee940d0d50 con 0x7fee84003f50 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8400baa0 con 0x7fee84004d10 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8400bc60 con 0x7fee9406ef40 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.114+0000 7fee93fff640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee84003f50 0x7fee840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee992b8640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84004d10 0x7fee84005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee93fff640 1 -- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 msgr2=0x7fee840039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee98ab7640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 0x7fee840039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee93fff640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 0x7fee840039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee93fff640 1 -- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84004d10 msgr2=0x7fee84005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:25.127 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee93fff640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84004d10 0x7fee84005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.127 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee93fff640 1 -- 172.21.15.28:0/1735781230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee84003530 con 0x7fee84003f50 2023-12-17T06:32:25.127 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee98ab7640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee9406ef40 0x7fee840039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-17T06:32:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee93fff640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee84003f50 0x7fee840047d0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fee80001ec0 tx=0x7fee8001f350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2da04784352bf93 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80005ae0 con 0x7fee84003f50 2023-12-17T06:32:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee800ab070 con 0x7fee84003f50 2023-12-17T06:32:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee937fe640 1 -- 172.21.15.28:0/1735781230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80005e80 con 0x7fee84003f50 2023-12-17T06:32:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee9ad1b640 1 -- 172.21.15.28:0/1735781230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee84003f50 msgr2=0x7fee840047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee9ad1b640 1 --2- 172.21.15.28:0/1735781230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee84003f50 0x7fee840047d0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fee80001ec0 tx=0x7fee8001f350 comp rx=0 tx=0).stop 2023-12-17T06:32:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee992b8640 1 -- 172.21.15.28:0/1735781230 reap_dead start 2023-12-17T06:32:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee9ad1b640 1 -- 172.21.15.28:0/1735781230 shutdown_connections 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee9ad1b640 1 -- 172.21.15.28:0/1735781230 >> 172.21.15.28:0/1735781230 conn(0x7fee940d0f10 msgr2=0x7fee940d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee9ad1b640 1 -- 172.21.15.28:0/1735781230 shutdown_connections 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.118+0000 7fee9ad1b640 1 -- 172.21.15.28:0/1735781230 wait complete. 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 Processor -- start 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 -- start start 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84003f50 0x7fee941f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9415cc80 0x7fee9413b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee941f1560 0x7fee940b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9415c280 con 0x7fee9415cc80 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9413b590 con 0x7fee84003f50 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9415b600 con 0x7fee941f1560 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9415cc80 0x7fee9413b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9415cc80 0x7fee9413b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36938/0 (socket says 172.21.15.28:36938) 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 -- 172.21.15.28:0/2668425668 learned_addr learned my addr 172.21.15.28:0/2668425668 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee98ab7640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84003f50 0x7fee941f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee992b8640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee941f1560 0x7fee940b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 -- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee941f1560 msgr2=0x7fee940b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fee941f1560 0x7fee940b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 -- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84003f50 msgr2=0x7fee941f0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84003f50 0x7fee941f0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 -- 172.21.15.28:0/2668425668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9415b8e0 con 0x7fee9415cc80 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee98ab7640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee84003f50 0x7fee941f0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee93fff640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9415cc80 0x7fee9413b050 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fee8001f390 tx=0x7fee84005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee800b7070 con 0x7fee9415cc80 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee80005940 con 0x7fee9415cc80 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9415bbc0 con 0x7fee9415cc80 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80005cd0 con 0x7fee9415cc80 2023-12-17T06:32:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee941f1a10 con 0x7fee9415cc80 2023-12-17T06:32:25.134 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fee9415c070 con 0x7fee9415cc80 2023-12-17T06:32:25.134 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fee800ad020 con 0x7fee9415cc80 2023-12-17T06:32:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fee78066470 0x7fee78068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fee80120b20 con 0x7fee9415cc80 2023-12-17T06:32:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7fee7806c2d0 0x7fee7806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 --> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fee7806eda0 con 0x7fee7806c2d0 2023-12-17T06:32:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee80120f70 con 0x7fee9415cc80 2023-12-17T06:32:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.122+0000 7fee98ab7640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fee78066470 0x7fee78068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.126+0000 7fee992b8640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7fee7806c2d0 0x7fee7806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.126+0000 7fee992b8640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7fee7806c2d0 0x7fee7806e6f0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.126+0000 7fee98ab7640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fee78066470 0x7fee78068930 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fee940b2ab0 tx=0x7fee88009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.126+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 <== osd.7 v2:172.21.15.122:6808/4051779013 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fee7806eda0 con 0x7fee7806c2d0 2023-12-17T06:32:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.138+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 --> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fee940cd3b0 con 0x7fee7806c2d0 2023-12-17T06:32:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.138+0000 7fee917fa640 1 -- 172.21.15.28:0/2668425668 <== osd.7 v2:172.21.15.122:6808/4051779013 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fee8c00f040 con 0x7fee7806c2d0 2023-12-17T06:32:25.147 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2023-12-17T06:32:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-17T06:32:25.147 INFO:teuthology.orchestra.run.smithi028.stdout:} 2023-12-17T06:32:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7fee7806c2d0 msgr2=0x7fee7806e6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6808/4051779013,v1:172.21.15.122:6809/4051779013] conn(0x7fee7806c2d0 0x7fee7806e6f0 crc :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fee78066470 msgr2=0x7fee78068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fee78066470 0x7fee78068930 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fee940b2ab0 tx=0x7fee88009040 comp rx=0 tx=0).stop 2023-12-17T06:32:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9415cc80 msgr2=0x7fee9413b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 --2- 172.21.15.28:0/2668425668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9415cc80 0x7fee9413b050 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fee8001f390 tx=0x7fee84005800 comp rx=0 tx=0).stop 2023-12-17T06:32:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee992b8640 1 -- 172.21.15.28:0/2668425668 reap_dead start 2023-12-17T06:32:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 shutdown_connections 2023-12-17T06:32:25.149 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 >> 172.21.15.28:0/2668425668 conn(0x7fee940d0f10 msgr2=0x7fee9406e830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:25.149 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 shutdown_connections 2023-12-17T06:32:25.149 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.142+0000 7fee9ad1b640 1 -- 172.21.15.28:0/2668425668 wait complete. 2023-12-17T06:32:25.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-17T06:32:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8cb5cc640 1 Processor -- start 2023-12-17T06:32:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8cb5cc640 1 -- start start 2023-12-17T06:32:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8cb5cc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8cb5cc640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd8c40cc7b0 0x7fd8c40b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8cb5cc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c40cc320 con 0x7fd8c40b0bb0 2023-12-17T06:32:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8cb5cc640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c40cfbf0 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8cb5cc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c40cbcd0 con 0x7fd8c415b230 2023-12-17T06:32:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c9368640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c8b67640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7fd8c40cc7b0 0x7fd8c40b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c9368640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37472/0 (socket says 172.21.15.28:37472) 2023-12-17T06:32:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c9368640 1 -- 172.21.15.28:0/1465165522 learned_addr learned my addr 172.21.15.28:0/1465165522 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c9368640 1 -- 172.21.15.28:0/1465165522 >> v2:172.21.15.122:3300/0 conn(0x7fd8c40cc7b0 msgr2=0x7fd8c40b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c9368640 1 --2- 172.21.15.28:0/1465165522 >> v2:172.21.15.122:3300/0 conn(0x7fd8c40cc7b0 0x7fd8c40b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c9368640 1 -- 172.21.15.28:0/1465165522 >> v1:172.21.15.122:6789/0 conn(0x7fd8c40b0bb0 legacy=0x7fd8c40999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c9368640 1 -- 172.21.15.28:0/1465165522 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c40cbfb0 con 0x7fd8c415b230 2023-12-17T06:32:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.266+0000 7fd8c8b67640 1 --2- 172.21.15.28:0/1465165522 >> v2:172.21.15.122:3300/0 conn(0x7fd8c40cc7b0 0x7fd8c40b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-17T06:32:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8c9368640 1 --2- 172.21.15.28:0/1465165522 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40cf510 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009870 tx=0x7fd8c001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f3025b5273935cc server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8bbfff640 1 -- 172.21.15.28:0/1465165522 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00ab600 con 0x7fd8c415b230 2023-12-17T06:32:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8bbfff640 1 -- 172.21.15.28:0/1465165522 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c00a7050 con 0x7fd8c415b230 2023-12-17T06:32:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8bbfff640 1 -- 172.21.15.28:0/1465165522 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00ab980 con 0x7fd8c415b230 2023-12-17T06:32:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/1465165522 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 msgr2=0x7fd8c40cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/1465165522 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40cf510 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009870 tx=0x7fd8c001c420 comp rx=0 tx=0).stop 2023-12-17T06:32:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/1465165522 shutdown_connections 2023-12-17T06:32:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/1465165522 >> v2:172.21.15.122:3300/0 conn(0x7fd8c40cc7b0 0x7fd8c40b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/1465165522 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40cf510 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/1465165522 >> 172.21.15.28:0/1465165522 conn(0x7fd8c40d0f30 msgr2=0x7fd8c40d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/1465165522 shutdown_connections 2023-12-17T06:32:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/1465165522 wait complete. 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 Processor -- start 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.270+0000 7fd8cb5cc640 1 -- start start 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c40b0bb0 0x7fd8c4152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c40cc7b0 0x7fd8c40b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c406c110 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c406c530 con 0x7fd8c40b0bb0 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c406d600 con 0x7fd8c415b230 2023-12-17T06:32:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c40cc7b0 0x7fd8c40b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c40cc7b0 0x7fd8c40b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36954/0 (socket says 172.21.15.28:36954) 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 -- 172.21.15.28:0/4264439123 learned_addr learned my addr 172.21.15.28:0/4264439123 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c9368640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c40b0bb0 0x7fd8c4152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 -- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 msgr2=0x7fd8c40b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 -- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c40b0bb0 msgr2=0x7fd8c4152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c40b0bb0 0x7fd8c4152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 -- 172.21.15.28:0/4264439123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4009420 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c8b67640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c40cc7b0 0x7fd8c40b5430 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400e9b0 tx=0x7fd8b400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8b9ffb640 1 -- 172.21.15.28:0/4264439123 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b400ca00 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8b9ffb640 1 -- 172.21.15.28:0/4264439123 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b400cb60 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c00094d0 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8b9ffb640 1 -- 172.21.15.28:0/4264439123 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4013470 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c406ba50 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c40d0950 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8b9ffb640 1 -- 172.21.15.28:0/4264439123 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7fd8b40135d0 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8b9ffb640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd8a8066470 0x7fd8a8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8b9ffb640 1 -- 172.21.15.28:0/4264439123 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd8b4083e20 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.274+0000 7fd8c9368640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd8a8066470 0x7fd8a8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.278+0000 7fd8c9368640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd8a8066470 0x7fd8a8068930 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00ad000 tx=0x7fd8c00a6000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.278+0000 7fd8b9ffb640 1 -- 172.21.15.28:0/4264439123 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7fd8b4051590 con 0x7fd8c40cc7b0 2023-12-17T06:32:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.430+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8c40cd430 con 0x7fd8a8066470 2023-12-17T06:32:25.438 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.430+0000 7fd8b9ffb640 1 -- 172.21.15.28:0/4264439123 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd8c40cd430 con 0x7fd8a8066470 2023-12-17T06:32:25.438 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 7 to deep-scrub 2023-12-17T06:32:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd8a8066470 msgr2=0x7fd8a8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd8a8066470 0x7fd8a8068930 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00ad000 tx=0x7fd8c00a6000 comp rx=0 tx=0).stop 2023-12-17T06:32:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c40cc7b0 msgr2=0x7fd8c40b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c40cc7b0 0x7fd8c40b5430 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400e9b0 tx=0x7fd8b400ee70 comp rx=0 tx=0).stop 2023-12-17T06:32:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 shutdown_connections 2023-12-17T06:32:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7fd8a8066470 0x7fd8a8068930 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7fd8c415b230 0x7fd8c40b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c40cc7b0 0x7fd8c40b5430 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 --2- 172.21.15.28:0/4264439123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c40b0bb0 0x7fd8c4152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 >> 172.21.15.28:0/4264439123 conn(0x7fd8c40d0f30 msgr2=0x7fd8c406fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 shutdown_connections 2023-12-17T06:32:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.434+0000 7fd8cb5cc640 1 -- 172.21.15.28:0/4264439123 wait complete. 2023-12-17T06:32:25.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-17T06:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.562+0000 7ff3b58fe640 1 Processor -- start 2023-12-17T06:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.562+0000 7ff3b58fe640 1 -- start start 2023-12-17T06:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3b58fe640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff3b00cf510 0x7ff3b00b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3b58fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00b0bb0 0x7ff3b015ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3b58fe640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b00cc320 con 0x7ff3b015b230 2023-12-17T06:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3b58fe640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b00cf9a0 con 0x7ff3b00cf510 2023-12-17T06:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3b58fe640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00cfbf0 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3af7fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00b0bb0 0x7ff3b015ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3a7fff640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff3b00cf510 0x7ff3b00b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3af7fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00b0bb0 0x7ff3b015ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3301/0 says I am v2:172.21.15.28:37500/0 (socket says 172.21.15.28:37500) 2023-12-17T06:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3af7fe640 1 -- 172.21.15.28:0/2906447385 learned_addr learned my addr 172.21.15.28:0/2906447385 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3af7fe640 1 -- 172.21.15.28:0/2906447385 >> v2:172.21.15.122:3300/0 conn(0x7ff3b00cf510 msgr2=0x7ff3b00b0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3af7fe640 1 --2- 172.21.15.28:0/2906447385 >> v2:172.21.15.122:3300/0 conn(0x7ff3b00cf510 0x7ff3b00b0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3af7fe640 1 -- 172.21.15.28:0/2906447385 >> v1:172.21.15.122:6789/0 conn(0x7ff3b015b230 legacy=0x7ff3b00cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3af7fe640 1 -- 172.21.15.28:0/2906447385 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b00cbcd0 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3af7fe640 1 --2- 172.21.15.28:0/2906447385 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00b0bb0 0x7ff3b015ec00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009e80 tx=0x7ff3a801c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a2fab36f6660601 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3ae7fc640 1 -- 172.21.15.28:0/2906447385 <== mon.2 v2:172.21.15.122:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80a56b0 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3ae7fc640 1 -- 172.21.15.28:0/2906447385 <== mon.2 v2:172.21.15.122:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a80a5810 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.566+0000 7ff3ae7fc640 1 -- 172.21.15.28:0/2906447385 <== mon.2 v2:172.21.15.122:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80a5ba0 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- 172.21.15.28:0/2906447385 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00b0bb0 msgr2=0x7ff3b015ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/2906447385 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00b0bb0 0x7ff3b015ec00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009e80 tx=0x7ff3a801c960 comp rx=0 tx=0).stop 2023-12-17T06:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- 172.21.15.28:0/2906447385 shutdown_connections 2023-12-17T06:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/2906447385 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00b0bb0 0x7ff3b015ec00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/2906447385 >> v2:172.21.15.122:3300/0 conn(0x7ff3b00cf510 0x7ff3b00b0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- 172.21.15.28:0/2906447385 >> 172.21.15.28:0/2906447385 conn(0x7ff3b00d0f30 msgr2=0x7ff3b00d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- 172.21.15.28:0/2906447385 shutdown_connections 2023-12-17T06:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- 172.21.15.28:0/2906447385 wait complete. 2023-12-17T06:32:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 Processor -- start 2023-12-17T06:32:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- start start 2023-12-17T06:32:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b00b0bb0 0x7ff3b0151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00cf510 0x7ff3b00b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3b015b230 0x7ff3b01525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00c91f0 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00c94d0 con 0x7ff3b015b230 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00c97b0 con 0x7ff3b00cf510 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b00b0bb0 0x7ff3b0151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b00b0bb0 0x7ff3b0151400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36970/0 (socket says 172.21.15.28:36970) 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 -- 172.21.15.28:0/1591272944 learned_addr learned my addr 172.21.15.28:0/1591272944 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3af7fe640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3b015b230 0x7ff3b01525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3a7fff640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00cf510 0x7ff3b00b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 -- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00cf510 msgr2=0x7ff3b00b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00cf510 0x7ff3b00b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 -- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3b015b230 msgr2=0x7ff3b01525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3b015b230 0x7ff3b01525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 -- 172.21.15.28:0/1591272944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a8009b30 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3a7fff640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00cf510 0x7ff3b00b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3aeffd640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b00b0bb0 0x7ff3b0151400 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c420 tx=0x7ff3a000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3acff9640 1 -- 172.21.15.28:0/1591272944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00052d0 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b00c9a90 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b0153280 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3acff9640 1 -- 172.21.15.28:0/1591272944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0005430 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3acff9640 1 -- 172.21.15.28:0/1591272944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0007260 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.570+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff394005260 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.574+0000 7ff3acff9640 1 -- 172.21.15.28:0/1591272944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7ff3a00073c0 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.574+0000 7ff3acff9640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff390066470 0x7ff390068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.574+0000 7ff3acff9640 1 -- 172.21.15.28:0/1591272944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff3a00837e0 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.574+0000 7ff3a7fff640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff390066470 0x7ff390068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.574+0000 7ff3a7fff640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff390066470 0x7ff390068930 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0153630 tx=0x7ff398008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.578+0000 7ff3acff9640 1 -- 172.21.15.28:0/1591272944 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7ff3a0050f60 con 0x7ff3b00b0bb0 2023-12-17T06:32:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.722+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff394000bd0 con 0x7ff390066470 2023-12-17T06:32:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3acff9640 1 -- 172.21.15.28:0/1591272944 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+49091 (secure 0 0 0) 0x7ff394000bd0 con 0x7ff390066470 2023-12-17T06:32:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:25.736 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-17T06:32:25.737 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff390066470 msgr2=0x7ff390068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff390066470 0x7ff390068930 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0153630 tx=0x7ff398008040 comp rx=0 tx=0).stop 2023-12-17T06:32:25.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b00b0bb0 msgr2=0x7ff3b0151400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:25.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b00b0bb0 0x7ff3b0151400 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c420 tx=0x7ff3a000c8e0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 shutdown_connections 2023-12-17T06:32:25.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff390066470 0x7ff390068930 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff3b015b230 0x7ff3b01525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff3b00cf510 0x7ff3b00b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.738 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 --2- 172.21.15.28:0/1591272944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b00b0bb0 0x7ff3b0151400 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 >> 172.21.15.28:0/1591272944 conn(0x7ff3b00d0f30 msgr2=0x7ff3b00b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 shutdown_connections 2023-12-17T06:32:25.739 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:25.730+0000 7ff3b58fe640 1 -- 172.21.15.28:0/1591272944 wait complete. 2023-12-17T06:32:25.753 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":15474,"stamp":"2023-12-17T06:32:24.774024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":14848760,"kb_used_data":1986936,"kb_used_omap":3694,"kb_used_meta":12858065,"kb_avail":740125960,"statfs":{"total":773094113280,"available":757888983040,"internally_reserved":0,"allocated":2034622464,"data_stored":2028064618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3783506,"internal_metadata":13166658734},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1658,"apply_latency_ms":1658,"commit_latency_ns":1658000000,"apply_latency_ns":1658000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003745"},"pg_stats":[{"pgid":"1.0","version":"36'161","reported_seq":485,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-12-17T06:32:11.845012+0000","last_change":"2023-12-16T21:57:32.506611+0000","last_active":"2023-12-17T06:32:11.845012+0000","last_peered":"2023-12-17T06:32:11.845012+0000","last_clean":"2023-12-17T06:32:11.845012+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-17T06:32:11.845012+0000","last_undegraded":"2023-12-17T06:32:11.845012+0000","last_fullsized":"2023-12-17T06:32:11.845012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:57:31.463438+0000","last_clean_scrub_stamp":"2023-12-16T21:57:31.463438+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:55:33.245348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719482917,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1675312,"kb_used_data":179880,"kb_used_omap":20,"kb_used_meta":1495403,"kb_avail":92696528,"statfs":{"total":96636764160,"available":94921244672,"internally_reserved":0,"allocated":184197120,"data_stored":183552500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20951,"internal_metadata":1531293225},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":140,"apply_latency_ms":140,"commit_latency_ns":140000000,"apply_latency_ns":140000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.9750000000000001,"5min":1.1080000000000001,"15min":0.88500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":15.872,"5min":15.872,"15min":15.872},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.355,"5min":1.238,"15min":0.96999999999999997},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":22.158000000000001,"5min":22.158000000000001,"15min":22.158000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.97099999999999997,"15min":0.82199999999999995},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":8.2279999999999998,"5min":8.2279999999999998,"15min":11.856},"last":5.8879999999999999},{"interface":"front","average":{"1min":2.6429999999999998,"5min":1.4390000000000001,"15min":1.1599999999999999},"min":{"1min":0.30199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":19.102,"5min":19.102,"15min":19.102},"last":5.9580000000000002}]},{"osd":2,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.327,"15min":1.196},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":19.332000000000001,"5min":19.332000000000001,"15min":19.332000000000001},"last":0.621},{"interface":"front","average":{"1min":2.593,"5min":1.518,"15min":1.2969999999999999},"min":{"1min":0.33600000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":15.898,"5min":15.898,"15min":15.898},"last":0.99199999999999999}]},{"osd":3,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":3.137,"5min":1.635,"15min":1.365},"min":{"1min":0.35399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":24.094999999999999,"5min":24.094999999999999,"15min":24.094999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":2.3330000000000002,"5min":1.5529999999999999,"15min":1.26},"min":{"1min":0.39300000000000002,"5min":0.32500000000000001,"15min":0.28999999999999998},"max":{"1min":15.837,"5min":15.837,"15min":15.837},"last":1.032}]},{"osd":4,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.5089999999999999,"5min":1.141,"15min":0.86399999999999999},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27100000000000002},"max":{"1min":16.614000000000001,"5min":16.614000000000001,"15min":16.614000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.2989999999999999,"5min":0.83599999999999997,"15min":0.78600000000000003},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":8.2699999999999996,"5min":8.2699999999999996,"15min":8.2699999999999996},"last":1.1040000000000001}]},{"osd":5,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":0.93100000000000005,"15min":0.88900000000000001},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":8.1969999999999992,"5min":8.1969999999999992,"15min":11.741},"last":1.0649999999999999},{"interface":"front","average":{"1min":2.8210000000000002,"5min":1.2929999999999999,"15min":1.024},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":27.901,"5min":27.901,"15min":27.901},"last":0.73999999999999999}]},{"osd":6,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.0860000000000001,"15min":0.90200000000000002},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":0.77400000000000002},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.0660000000000001,"15min":0.95099999999999996},"min":{"1min":0.36299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.946,"5min":11.946,"15min":11.946},"last":0.84499999999999997}]}]},{"osd":6,"up_from":16,"seq":68719482917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1771128,"kb_used_data":432,"kb_used_omap":981,"kb_used_meta":1769706,"kb_avail":92600712,"statfs":{"total":96636764160,"available":94823129088,"internally_reserved":0,"allocated":442368,"data_stored":173304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1005312,"internal_metadata":1812179200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":581,"apply_latency_ms":581,"commit_latency_ns":581000000,"apply_latency_ns":581000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2879999999999998,"5min":1.2549999999999999,"15min":0.88900000000000001},"min":{"1min":0.26600000000000001,"5min":0.24399999999999999,"15min":0.22500000000000001},"max":{"1min":38.720999999999997,"5min":38.720999999999997,"15min":38.720999999999997},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.0669999999999999,"15min":0.90200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26500000000000001,"15min":0.25},"max":{"1min":5.3920000000000003,"5min":8.0779999999999994,"15min":8.0779999999999994},"last":0.64800000000000002}]},{"osd":1,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.1850000000000001,"15min":1.1990000000000001},"min":{"1min":0.26800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.2619999999999996,"5min":5.077,"15min":8.9410000000000007},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.639,"5min":1.1419999999999999,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":4.7590000000000003,"5min":8.798,"15min":8.798},"last":0.496}]},{"osd":2,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.3819999999999999,"15min":1.167},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.5819999999999999,"5min":8.9090000000000007,"15min":8.9090000000000007},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.496,"5min":1.425,"15min":1.236},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.29599999999999999},"max":{"1min":4.9790000000000001,"5min":8.8179999999999996,"15min":10.401999999999999},"last":1.0369999999999999}]},{"osd":3,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2029999999999998,"5min":1.8009999999999999,"15min":1.369},"min":{"1min":0.38700000000000001,"5min":0.31,"15min":0.31},"max":{"1min":16.004000000000001,"5min":16.004000000000001,"15min":16.004000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.339,"5min":1.1120000000000001,"15min":1.2470000000000001},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.32000000000000001},"max":{"1min":4.867,"5min":8.2040000000000006,"15min":8.2040000000000006},"last":0.98799999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.044,"15min":0.89000000000000001},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.25900000000000001},"max":{"1min":20.827000000000002,"5min":20.827000000000002,"15min":20.827000000000002},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0169999999999999,"15min":0.82899999999999996},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":10.022,"5min":10.022,"15min":10.022},"last":0.85699999999999998}]},{"osd":5,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.8079999999999998,"5min":1.448,"15min":1.073},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":41.658000000000001,"5min":41.658000000000001,"15min":41.658000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":3.9060000000000001,"5min":1.423,"15min":1.0580000000000001},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.29199999999999998},"max":{"1min":41.634999999999998,"5min":41.634999999999998,"15min":41.634999999999998},"last":0.94899999999999995}]},{"osd":7,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.012,"15min":0.93700000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":4.7960000000000003,"5min":5.2859999999999996,"15min":5.625},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.82499999999999996,"15min":0.874},"min":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.318},"max":{"1min":3.835,"5min":4.5359999999999996,"15min":7.0810000000000004},"last":0.89000000000000001}]}]},{"osd":5,"up_from":16,"seq":68719482917,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1689648,"kb_used_data":46696,"kb_used_omap":463,"kb_used_meta":1642480,"kb_avail":92682192,"statfs":{"total":96636764160,"available":94906564608,"internally_reserved":0,"allocated":47816704,"data_stored":47467506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475086,"internal_metadata":1681899570},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":116,"apply_latency_ms":116,"commit_latency_ns":116000000,"apply_latency_ns":116000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.66300000000000003,"15min":0.73799999999999999},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.23200000000000001},"max":{"1min":5.2619999999999996,"5min":5.2619999999999996,"15min":6.2859999999999996},"last":0.432},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.90700000000000003,"15min":0.84299999999999997},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":3.6600000000000001,"5min":5.5810000000000004,"15min":6.4829999999999997},"last":0.71299999999999997}]},{"osd":1,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.0549999999999999,"15min":1.018},"min":{"1min":0.27700000000000002,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":5.8109999999999999,"5min":9.2159999999999993,"15min":9.2159999999999993},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.92500000000000004,"15min":0.875},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":8.6959999999999997},"last":0.59399999999999997}]},{"osd":2,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.92700000000000005,"15min":0.90200000000000002},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28699999999999998},"max":{"1min":5.0919999999999996,"5min":5.4290000000000003,"15min":6.3460000000000001},"last":0.747},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.196,"15min":1.145},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":5.8810000000000002,"5min":5.8810000000000002,"15min":6.3540000000000001},"last":0.63300000000000001}]},{"osd":3,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":1.181,"15min":1.1160000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.307},"max":{"1min":20.481000000000002,"5min":20.481000000000002,"15min":20.481000000000002},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.415,"5min":1.25,"15min":1.1870000000000001},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.29099999999999998},"max":{"1min":5.8520000000000003,"5min":5.8520000000000003,"15min":6.444},"last":0.51200000000000001}]},{"osd":4,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.66900000000000004,"15min":0.68899999999999995},"min":{"1min":0.314,"5min":0.249,"15min":0.249},"max":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":6.2149999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79700000000000004,"15min":0.78400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.3490000000000002,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":0.84999999999999998}]},{"osd":6,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":1.1319999999999999,"15min":0.98699999999999999},"min":{"1min":0.38700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":27.823,"5min":27.823,"15min":27.823},"last":0.56599999999999995},{"interface":"front","average":{"1min":2.5529999999999999,"5min":1.0109999999999999,"15min":0.86199999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":33.023000000000003,"5min":33.023000000000003,"15min":33.023000000000003},"last":0.65800000000000003}]},{"osd":7,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.5230000000000001,"5min":1.4139999999999999,"15min":1.0029999999999999},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":42.939,"5min":42.939,"15min":42.939},"last":0.78300000000000003},{"interface":"front","average":{"1min":2.234,"5min":1.0629999999999999,"15min":1.0209999999999999},"min":{"1min":0.44600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":18.196999999999999,"5min":18.196999999999999,"15min":18.196999999999999},"last":0.76300000000000001}]}]},{"osd":4,"up_from":16,"seq":68719482915,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1740516,"kb_used_data":35548,"kb_used_omap":362,"kb_used_meta":1704597,"kb_avail":92631324,"statfs":{"total":96636764160,"available":94854475776,"internally_reserved":0,"allocated":36401152,"data_stored":35994641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":371446,"internal_metadata":1745507594},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":128,"apply_latency_ms":128,"commit_latency_ns":128000000,"apply_latency_ns":128000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.57599999999999996,"15min":0.69199999999999995},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.246},"max":{"1min":1.2909999999999999,"5min":4.8220000000000001,"15min":6.8559999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.244,"5min":0.94799999999999995,"15min":0.91400000000000003},"min":{"1min":0.309,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.5,"5min":7.1079999999999997,"15min":8.9420000000000002},"last":0.55500000000000005}]},{"osd":1,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.0249999999999999,"15min":1.0489999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.251},"max":{"1min":6.5339999999999998,"5min":7.3399999999999999,"15min":7.3399999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.81100000000000005,"15min":1.161},"min":{"1min":0.28699999999999998,"5min":0.27300000000000002,"15min":0.25900000000000001},"max":{"1min":3.6709999999999998,"5min":7.1849999999999996,"15min":27.398},"last":0.58999999999999997}]},{"osd":2,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.95899999999999996,"15min":0.98499999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":5.2569999999999997,"5min":7.9189999999999996,"15min":7.9189999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.964,"5min":1.5229999999999999,"15min":1.4219999999999999},"min":{"1min":0.41299999999999998,"5min":0.307,"15min":0.29299999999999998},"max":{"1min":6.641,"5min":10.103,"15min":10.103},"last":0.91100000000000003}]},{"osd":3,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.234,"15min":1.333},"min":{"1min":0.33700000000000002,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":5.1349999999999998,"5min":11.417999999999999,"15min":12.627000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.879,"5min":1.0629999999999999,"15min":1.2110000000000001},"min":{"1min":0.32400000000000001,"5min":0.318,"15min":0.318},"max":{"1min":4.2089999999999996,"5min":8.1259999999999994,"15min":10.43},"last":0.65500000000000003}]},{"osd":5,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.81599999999999995,"15min":0.78000000000000003},"min":{"1min":0.28599999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":6.4660000000000002,"5min":6.4660000000000002,"15min":8.7249999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.61199999999999999,"15min":0.73899999999999999},"min":{"1min":0.33700000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.105,"5min":4.2629999999999999,"15min":6.6559999999999997},"last":0.44500000000000001}]},{"osd":6,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.72199999999999998,"15min":0.80100000000000005},"min":{"1min":0.34699999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.1990000000000001,"5min":7.5999999999999996,"15min":7.5999999999999996},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.004,"15min":0.91000000000000003},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.34000000000000002},"max":{"1min":6.4249999999999998,"5min":6.4249999999999998,"15min":6.4249999999999998},"last":0.85999999999999999}]},{"osd":7,"last update":"Sun Dec 17 06:31:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.74199999999999999,"15min":0.84499999999999997},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":6.1219999999999999,"5min":6.1219999999999999,"15min":6.6879999999999997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.88400000000000001,"15min":0.91300000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.80000000000000004,"5min":5.2919999999999998,"15min":6.9379999999999997},"last":0.88200000000000001}]}]},{"osd":3,"up_from":15,"seq":64424515609,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2430920,"kb_used_data":710016,"kb_used_omap":132,"kb_used_meta":1720763,"kb_avail":91940920,"statfs":{"total":96636764160,"available":94147502080,"internally_reserved":0,"allocated":727056384,"data_stored":725273036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136113,"internal_metadata":1762061391},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":120,"apply_latency_ms":120,"commit_latency_ns":120000000,"apply_latency_ns":120000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":1.1759999999999999,"15min":1.1319999999999999},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.20000000000000001},"max":{"1min":21.396999999999998,"5min":21.396999999999998,"15min":21.396999999999998},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.393,"5min":0.93799999999999994,"15min":0.92000000000000004},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":12.132,"5min":12.132,"15min":12.132},"last":0.76500000000000001}]},{"osd":1,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":1.339,"15min":1.1479999999999999},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.215},"max":{"1min":29.577000000000002,"5min":29.577000000000002,"15min":29.577000000000002},"last":0.999},{"interface":"front","average":{"1min":2.0870000000000002,"5min":1.4950000000000001,"15min":1.373},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":12.877000000000001,"5min":12.877000000000001,"15min":12.877000000000001},"last":0.83599999999999997}]},{"osd":2,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":1.3200000000000001,"15min":1.395},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.253},"max":{"1min":12.920999999999999,"5min":12.920999999999999,"15min":12.920999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.3380000000000001,"15min":1.177},"min":{"1min":0.34999999999999998,"5min":0.29799999999999999,"15min":0.27500000000000002},"max":{"1min":17.669,"5min":17.669,"15min":17.669},"last":1.0589999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.6339999999999999,"5min":1.488,"15min":1.2769999999999999},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":13.095000000000001,"5min":13.228,"15min":13.228},"last":0.87},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.161,"15min":1.0860000000000001},"min":{"1min":0.35899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":12.166,"5min":12.166,"15min":12.166},"last":0.81299999999999994}]},{"osd":5,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.3889999999999998,"5min":1.4710000000000001,"15min":1.234},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":29.622,"5min":29.622,"15min":29.622},"last":0.84699999999999998},{"interface":"front","average":{"1min":3.5619999999999998,"5min":1.6699999999999999,"15min":1.409},"min":{"1min":0.38100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":21.038,"5min":21.038,"15min":21.038},"last":0.94699999999999995}]},{"osd":6,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.1539999999999999,"15min":1.105},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.34999999999999998},"max":{"1min":4.0380000000000003,"5min":5.5220000000000002,"15min":10.683},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.786,"5min":1.296,"15min":1.1779999999999999},"min":{"1min":0.47299999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":15.619999999999999,"5min":15.619999999999999,"15min":15.619999999999999},"last":0.89000000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.2989999999999999,"5min":1.71,"15min":1.4810000000000001},"min":{"1min":0.42099999999999999,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":18.16,"5min":18.16,"15min":18.16},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.643,"15min":1.288},"min":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.41799999999999998},"max":{"1min":10.282999999999999,"5min":22.334,"15min":22.334},"last":1.0189999999999999}]}]},{"osd":1,"up_from":15,"seq":64424515614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1852152,"kb_used_data":432,"kb_used_omap":997,"kb_used_meta":1850714,"kb_avail":92519688,"statfs":{"total":96636764160,"available":94740160512,"internally_reserved":0,"allocated":442368,"data_stored":173304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021567,"internal_metadata":1895131521},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":284,"apply_latency_ms":284,"commit_latency_ns":284000000,"apply_latency_ns":284000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.81499999999999995,"15min":0.89000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.5490000000000004,"5min":9.8010000000000002,"15min":9.8010000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.089,"15min":1.196},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.214},"max":{"1min":30.109999999999999,"5min":30.109999999999999,"15min":30.109999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5129999999999999,"5min":1.2869999999999999,"15min":1.202},"min":{"1min":0.30499999999999999,"5min":0.23300000000000001,"15min":0.224},"max":{"1min":25.259,"5min":25.259,"15min":25.259},"last":0.627},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.873,"15min":0.998},"min":{"1min":0.29199999999999998,"5min":0.245,"15min":0.22500000000000001},"max":{"1min":5.4580000000000002,"5min":9.7720000000000002,"15min":9.7720000000000002},"last":0.68700000000000006}]},{"osd":3,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.8199999999999998,"5min":1.218,"15min":1.2929999999999999},"min":{"1min":0.314,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":30.460999999999999,"5min":30.460999999999999,"15min":30.460999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.8120000000000001,"5min":1.4039999999999999,"15min":1.3080000000000001},"min":{"1min":0.311,"5min":0.28999999999999998,"15min":0.25},"max":{"1min":6.6210000000000004,"5min":8.0980000000000008,"15min":9.6460000000000008},"last":0.66900000000000004}]},{"osd":4,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1519999999999999,"15min":0.96899999999999997},"min":{"1min":0.34999999999999998,"5min":0.31,"15min":0.27300000000000002},"max":{"1min":5.5190000000000001,"5min":9.6669999999999998,"15min":10.122},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.0780000000000001,"15min":1.0600000000000001},"min":{"1min":0.378,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":0.80100000000000005}]},{"osd":5,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.127,"15min":0.99199999999999999},"min":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.315},"max":{"1min":5.492,"5min":9.7110000000000003,"15min":9.7110000000000003},"last":0.74299999999999999},{"interface":"front","average":{"1min":2.851,"5min":1.218,"15min":1.2490000000000001},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":30.277999999999999,"5min":30.277999999999999,"15min":30.277999999999999},"last":0.70999999999999996}]},{"osd":6,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.1830000000000001,"15min":1.1819999999999999},"min":{"1min":0.46700000000000003,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":4.8200000000000003,"5min":6.3979999999999997,"15min":8.8840000000000003},"last":0.72799999999999998},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.5109999999999999,"15min":1.1779999999999999},"min":{"1min":0.435,"5min":0.39100000000000001,"15min":0.38700000000000001},"max":{"1min":19.016999999999999,"5min":19.016999999999999,"15min":19.016999999999999},"last":0.83299999999999996}]},{"osd":7,"last update":"Sun Dec 17 06:31:26 2023","interfaces":[{"interface":"back","average":{"1min":2.9620000000000002,"5min":1.401,"15min":1.284},"min":{"1min":0.52600000000000002,"5min":0.40699999999999997,"15min":0.378},"max":{"1min":30.242999999999999,"5min":30.242999999999999,"15min":30.242999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.659,"5min":1.2869999999999999,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":4.9240000000000004,"5min":6.46,"15min":8.9589999999999996},"last":0.76500000000000001}]}]},{"osd":0,"up_from":15,"seq":64424515616,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1963104,"kb_used_data":564504,"kb_used_omap":422,"kb_used_meta":1398169,"kb_avail":92408736,"statfs":{"total":96636764160,"available":94626545664,"internally_reserved":0,"allocated":578052096,"data_stored":576545262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":432402,"internal_metadata":1431725806},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":0.88400000000000001,"15min":0.97199999999999998},"min":{"1min":0.20200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":8.3789999999999996,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.96199999999999997,"15min":1.181},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1079999999999997,"5min":5.1470000000000002,"15min":8.1039999999999992},"last":2.3149999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.159,"15min":1.194},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.2759999999999998,"5min":7.2759999999999998,"15min":10.146000000000001},"last":2.4990000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.2130000000000001,"15min":1.2889999999999999},"min":{"1min":0.26600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.4770000000000003,"5min":13.676,"15min":13.676},"last":2.9039999999999999}]},{"osd":3,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.2929999999999999,"15min":1.2749999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.25},"max":{"1min":2.5680000000000001,"5min":13.721,"15min":13.721},"last":2.9740000000000002},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.0840000000000001,"15min":1.1870000000000001},"min":{"1min":0.37,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":6.2089999999999996,"5min":8.0570000000000004,"15min":10.037000000000001},"last":0.55600000000000005}]},{"osd":4,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.145,"15min":1.1919999999999999},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.0579999999999998,"5min":13.755000000000001,"15min":13.755000000000001},"last":2.8410000000000002},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.0389999999999999,"15min":0.93200000000000005},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.29199999999999998},"max":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":7.3540000000000001},"last":2.6789999999999998}]},{"osd":5,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":0.95499999999999996,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.32700000000000001},"max":{"1min":6.5090000000000003,"5min":6.5090000000000003,"15min":7.4349999999999996},"last":1.8640000000000001},{"interface":"front","average":{"1min":1.044,"5min":1.0089999999999999,"15min":1.149},"min":{"1min":0.379,"5min":0.31,"15min":0.31},"max":{"1min":4.1589999999999998,"5min":5.101,"15min":8.1820000000000004},"last":2.3900000000000001}]},{"osd":6,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.25,"15min":1.3069999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.34300000000000003},"max":{"1min":6.202,"5min":13.819000000000001,"15min":13.819000000000001},"last":5.8899999999999997},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.1499999999999999,"15min":1.087},"min":{"1min":0.38,"5min":0.38,"15min":0.378},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":5.8479999999999999}]},{"osd":7,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.087,"15min":1.2030000000000001},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":4.2119999999999997,"5min":5.1760000000000002,"15min":8.2110000000000003},"last":2.5459999999999998},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.266,"15min":1.4059999999999999},"min":{"1min":0.39500000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.242,"5min":13.786,"15min":13.786},"last":2.9399999999999999}]}]},{"osd":2,"up_from":14,"seq":60129548317,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1725980,"kb_used_data":449428,"kb_used_omap":313,"kb_used_meta":1276230,"kb_avail":92645860,"statfs":{"total":96636764160,"available":94869360640,"internally_reserved":0,"allocated":460214272,"data_stored":458885065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":320629,"internal_metadata":1306860427},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":172,"apply_latency_ms":172,"commit_latency_ns":172000000,"apply_latency_ns":172000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.3380000000000001,"5min":7.5919999999999996,"15min":8.1500000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.22,"15min":1.125},"min":{"1min":0.22,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":4.0030000000000001,"5min":10.301,"15min":10.301},"last":0.59299999999999997}]},{"osd":1,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.98099999999999998,"15min":1.0920000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.048,"5min":6.8010000000000002,"15min":7.7320000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.218,"15min":1.2370000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.9859999999999998,"5min":10.345000000000001,"15min":10.345000000000001},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.482,"15min":1.411},"min":{"1min":0.27400000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.3250000000000002,"5min":10.706,"15min":10.706},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.216,"15min":1.1899999999999999},"min":{"1min":0.32900000000000001,"5min":0.29699999999999999,"15min":0.26100000000000001},"max":{"1min":14.513999999999999,"5min":14.513999999999999,"15min":14.513999999999999},"last":0.51700000000000002}]},{"osd":4,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.073,"15min":1.073},"min":{"1min":0.39000000000000001,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":3.1899999999999999,"5min":6.1870000000000003,"15min":8.0860000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.399,"5min":1.276,"15min":1.1679999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":10.583,"5min":10.583,"15min":10.583},"last":0.63700000000000001}]},{"osd":5,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":1.6859999999999999,"15min":1.383},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":30.448,"5min":30.448,"15min":30.448},"last":0.70399999999999996},{"interface":"front","average":{"1min":2.6619999999999999,"5min":1.6459999999999999,"15min":1.4610000000000001},"min":{"1min":0.38500000000000001,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":21.797999999999998,"5min":21.797999999999998,"15min":21.797999999999998},"last":0.503}]},{"osd":6,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.758,"15min":1.5089999999999999},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.376},"max":{"1min":15.653,"5min":15.653,"15min":15.653},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.329,"15min":1.2290000000000001},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.35999999999999999},"max":{"1min":15.82,"5min":15.82,"15min":15.82},"last":0.76600000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.4770000000000001,"15min":1.335},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.37},"max":{"1min":8.0250000000000004,"5min":9.5489999999999995,"15min":9.5489999999999995},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.508,"15min":1.47},"min":{"1min":0.44500000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":6.431,"5min":7.9429999999999996,"15min":10.446},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-17T06:32:25.757 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-16T21:57:31.463438+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=21, tm_min=57, tm_sec=31, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=17, tm_hour=6, tm_min=32, tm_sec=20, tm_wday=6, tm_yday=351, tm_isdst=0) 2023-12-17T06:32:25.757 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-17T06:32:45.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-17T06:32:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8143a45640 1 Processor -- start 2023-12-17T06:32:45.895 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8143a45640 1 -- start start 2023-12-17T06:32:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8143a45640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c06c2c0 0x7f813c06c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8143a45640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f813c0b3400 0x7f813c0b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8143a45640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c1408c0 con 0x7f813c140180 2023-12-17T06:32:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8143a45640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c06ccb0 con 0x7f813c0b3400 2023-12-17T06:32:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8143a45640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c15db80 con 0x7f813c06c2c0 2023-12-17T06:32:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f81417e1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f813c140180 0x7f813c071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45584/0 (socket says 172.21.15.28:45584) 2023-12-17T06:32:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f81417e1640 1 -- 172.21.15.28:0/4217820673 learned_addr learned my addr 172.21.15.28:0/4217820673 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8141fe2640 1 --2- 172.21.15.28:0/4217820673 >> v2:172.21.15.122:3300/0 conn(0x7f813c0b3400 0x7f813c0b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8140fe0640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c06c2c0 0x7f813c06c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553445471 0 0) 0x7f813c1408c0 con 0x7f813c140180 2023-12-17T06:32:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8128003530 con 0x7f813c140180 2023-12-17T06:32:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2394117972 0 0) 0x7f8128003530 con 0x7f813c140180 2023-12-17T06:32:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c06c2c0 msgr2=0x7f813c06c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c06c2c0 0x7f813c06c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:45.899 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 >> v2:172.21.15.122:3300/0 conn(0x7f813c0b3400 msgr2=0x7f813c0b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 --2- 172.21.15.28:0/4217820673 >> v2:172.21.15.122:3300/0 conn(0x7f813c0b3400 0x7f813c0b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c15de60 con 0x7f813c140180 2023-12-17T06:32:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7f8138001680 con 0x7f813c140180 2023-12-17T06:32:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 >> v1:172.21.15.28:6789/0 conn(0x7f813c140180 legacy=0x7f813c071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f8128003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81280047d0 0x7f8128004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c06c2c0 0x7f8128005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c06ccb0 con 0x7f8128003980 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c15db80 con 0x7f81280047d0 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f812800b9c0 con 0x7f813c06c2c0 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f81417e1640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f8128003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8141fe2640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c06c2c0 0x7f8128005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f8140fe0640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81280047d0 0x7f8128004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f81417e1640 1 -- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c06c2c0 msgr2=0x7f8128005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f81417e1640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c06c2c0 0x7f8128005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f81417e1640 1 -- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81280047d0 msgr2=0x7f8128004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:45.901 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f81417e1640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81280047d0 0x7f8128004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:45.902 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.889+0000 7f81417e1640 1 -- 172.21.15.28:0/4217820673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8128003530 con 0x7f8128003980 2023-12-17T06:32:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f81417e1640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f8128003da0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f813800c290 tx=0x7f813801cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73102fd88b2b5c5a server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c0b4580 con 0x7f8128003980 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81380058e0 con 0x7f8128003980 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8133fff640 1 -- 172.21.15.28:0/4217820673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138005c90 con 0x7f8128003980 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8143a45640 1 -- 172.21.15.28:0/4217820673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 msgr2=0x7f8128003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8143a45640 1 --2- 172.21.15.28:0/4217820673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f8128003da0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f813800c290 tx=0x7f813801cc50 comp rx=0 tx=0).stop 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8141fe2640 1 -- 172.21.15.28:0/4217820673 reap_dead start 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8143a45640 1 -- 172.21.15.28:0/4217820673 shutdown_connections 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8143a45640 1 -- 172.21.15.28:0/4217820673 >> 172.21.15.28:0/4217820673 conn(0x7f813c06a530 msgr2=0x7f813c06a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8143a45640 1 -- 172.21.15.28:0/4217820673 shutdown_connections 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8143a45640 1 -- 172.21.15.28:0/4217820673 wait complete. 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8143a45640 1 Processor -- start 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.893+0000 7f8143a45640 1 -- start start 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f813c130800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f813c161c30 0x7f813c1419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c141ef0 0x7f813c1628c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c142370 con 0x7f8128003980 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1620b0 con 0x7f813c161c30 2023-12-17T06:32:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c13bc10 con 0x7f813c141ef0 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f813c130800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f813c130800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33554/0 (socket says 172.21.15.28:33554) 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 -- 172.21.15.28:0/3351396473 learned_addr learned my addr 172.21.15.28:0/3351396473 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8141fe2640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c141ef0 0x7f813c1628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8140fe0640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f813c161c30 0x7f813c1419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 -- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c141ef0 msgr2=0x7f813c1628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c141ef0 0x7f813c1628c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 -- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f813c161c30 msgr2=0x7f813c1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f813c161c30 0x7f813c1419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 -- 172.21.15.28:0/3351396473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c1634a0 con 0x7f8128003980 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8141fe2640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c141ef0 0x7f813c1628c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f81417e1640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f813c130800 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f81380ab040 tx=0x7f8128005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8131ffb640 1 -- 172.21.15.28:0/3351396473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138005e70 con 0x7f8128003980 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8131ffb640 1 -- 172.21.15.28:0/3351396473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81380047f0 con 0x7f8128003980 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c160e70 con 0x7f8128003980 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8131ffb640 1 -- 172.21.15.28:0/3351396473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138004bc0 con 0x7f8128003980 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c161430 con 0x7f8128003980 2023-12-17T06:32:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c071750 con 0x7f8128003980 2023-12-17T06:32:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8131ffb640 1 -- 172.21.15.28:0/3351396473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f8138004d20 con 0x7f8128003980 2023-12-17T06:32:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8131ffb640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8120066470 0x7f8120068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8131ffb640 1 -- 172.21.15.28:0/3351396473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f813811d740 con 0x7f8128003980 2023-12-17T06:32:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.897+0000 7f8140fe0640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8120066470 0x7f8120068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.901+0000 7f8140fe0640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8120066470 0x7f8120068930 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f813c13b8a0 tx=0x7f812c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:45.910 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:45.901+0000 7f8131ffb640 1 -- 172.21.15.28:0/3351396473 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f81380eaf50 con 0x7f8128003980 2023-12-17T06:32:46.061 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.053+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 --> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f813c15d850 con 0x7f8120066470 2023-12-17T06:32:46.068 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8131ffb640 1 -- 172.21.15.28:0/3351396473 <== mgr.4114 v2:172.21.15.122:6832/3228142796 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48337 (secure 0 0 0) 0x7f812c012950 con 0x7f8120066470 2023-12-17T06:32:46.068 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:46.069 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2023-12-17T06:32:46.070 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8120066470 msgr2=0x7f8120068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8120066470 0x7f8120068930 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f813c13b8a0 tx=0x7f812c008040 comp rx=0 tx=0).stop 2023-12-17T06:32:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 msgr2=0x7f813c130800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f813c130800 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f81380ab040 tx=0x7f8128005800 comp rx=0 tx=0).stop 2023-12-17T06:32:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 shutdown_connections 2023-12-17T06:32:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f8120066470 0x7f8120068930 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f813c141ef0 0x7f813c1628c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f813c161c30 0x7f813c1419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.071 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 --2- 172.21.15.28:0/3351396473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8128003980 0x7f813c130800 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 >> 172.21.15.28:0/3351396473 conn(0x7f813c06a530 msgr2=0x7f813c15c230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:46.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 shutdown_connections 2023-12-17T06:32:46.072 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.061+0000 7f8143a45640 1 -- 172.21.15.28:0/3351396473 wait complete. 2023-12-17T06:32:46.086 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":15484,"stamp":"2023-12-17T06:32:44.779428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":12939080,"kb_used_data":4616,"kb_used_omap":3694,"kb_used_meta":12930705,"kb_avail":742035640,"statfs":{"total":773094113280,"available":759844495360,"internally_reserved":0,"allocated":4726784,"data_stored":2567168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3783506,"internal_metadata":13241042094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003059"},"pg_stats":[{"pgid":"1.0","version":"39'163","reported_seq":497,"reported_epoch":39,"state":"active+clean","last_fresh":"2023-12-17T06:32:37.518411+0000","last_change":"2023-12-17T06:32:37.518465+0000","last_active":"2023-12-17T06:32:37.518411+0000","last_peered":"2023-12-17T06:32:37.518411+0000","last_clean":"2023-12-17T06:32:37.518411+0000","last_became_active":"2023-12-16T21:57:32.504532+0000","last_became_peered":"2023-12-16T21:57:32.504532+0000","last_unstale":"2023-12-17T06:32:37.518411+0000","last_undegraded":"2023-12-17T06:32:37.518411+0000","last_fullsized":"2023-12-17T06:32:37.518411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"39'163","last_scrub_stamp":"2023-12-17T06:32:37.518292+0000","last_deep_scrub":"39'163","last_deep_scrub_stamp":"2023-12-17T06:32:37.518292+0000","last_clean_scrub_stamp":"2023-12-17T06:32:37.518292+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T10:35:48.045596+0000","scrub_duration":2.048018721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":16,"seq":68719482921,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1514492,"kb_used_data":1012,"kb_used_omap":20,"kb_used_meta":1513451,"kb_avail":92857348,"statfs":{"total":96636764160,"available":95085924352,"internally_reserved":0,"allocated":1036288,"data_stored":763672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20951,"internal_metadata":1549774377},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.9750000000000001,"5min":1.1080000000000001,"15min":0.88500000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":15.872,"5min":15.872,"15min":15.872},"last":0.71499999999999997},{"interface":"front","average":{"1min":2.355,"5min":1.238,"15min":0.96999999999999997},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.247},"max":{"1min":22.158000000000001,"5min":22.158000000000001,"15min":22.158000000000001},"last":1.0860000000000001}]},{"osd":1,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.766,"5min":0.97099999999999997,"15min":0.82199999999999995},"min":{"1min":0.29699999999999999,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":8.2279999999999998,"5min":8.2279999999999998,"15min":11.856},"last":1.0509999999999999},{"interface":"front","average":{"1min":2.6429999999999998,"5min":1.4390000000000001,"15min":1.1599999999999999},"min":{"1min":0.30199999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":19.102,"5min":19.102,"15min":19.102},"last":0.76200000000000001}]},{"osd":2,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.4300000000000002,"5min":1.327,"15min":1.196},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":19.332000000000001,"5min":19.332000000000001,"15min":19.332000000000001},"last":1.4159999999999999},{"interface":"front","average":{"1min":2.593,"5min":1.518,"15min":1.2969999999999999},"min":{"1min":0.33600000000000002,"5min":0.315,"15min":0.30399999999999999},"max":{"1min":15.898,"5min":15.898,"15min":15.898},"last":0.96399999999999997}]},{"osd":3,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":3.137,"5min":1.635,"15min":1.365},"min":{"1min":0.35399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":24.094999999999999,"5min":24.094999999999999,"15min":24.094999999999999},"last":1.248},{"interface":"front","average":{"1min":2.3330000000000002,"5min":1.5529999999999999,"15min":1.26},"min":{"1min":0.39300000000000002,"5min":0.32500000000000001,"15min":0.28999999999999998},"max":{"1min":15.837,"5min":15.837,"15min":15.837},"last":0.86399999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":2.5089999999999999,"5min":1.141,"15min":0.86399999999999999},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27100000000000002},"max":{"1min":16.614000000000001,"5min":16.614000000000001,"15min":16.614000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":0.83599999999999997,"15min":0.78600000000000003},"min":{"1min":0.315,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":8.2699999999999996,"5min":8.2699999999999996,"15min":8.2699999999999996},"last":1.1259999999999999}]},{"osd":5,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":0.93100000000000005,"15min":0.88900000000000001},"min":{"1min":0.36599999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":8.1969999999999992,"5min":8.1969999999999992,"15min":11.741},"last":1.1850000000000001},{"interface":"front","average":{"1min":2.8210000000000002,"5min":1.2929999999999999,"15min":1.024},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32300000000000001},"max":{"1min":27.901,"5min":27.901,"15min":27.901},"last":1.2110000000000001}]},{"osd":6,"last update":"Sun Dec 17 06:31:54 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.0860000000000001,"15min":0.90200000000000002},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.28399999999999997},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":1.286},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.0660000000000001,"15min":0.95099999999999996},"min":{"1min":0.36299999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.946,"5min":11.946,"15min":11.946},"last":1.3380000000000001}]}]},{"osd":6,"up_from":16,"seq":68719482921,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1771128,"kb_used_data":432,"kb_used_omap":981,"kb_used_meta":1769706,"kb_avail":92600712,"statfs":{"total":96636764160,"available":94823129088,"internally_reserved":0,"allocated":442368,"data_stored":173304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1005312,"internal_metadata":1812179200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2879999999999998,"5min":1.2549999999999999,"15min":0.88900000000000001},"min":{"1min":0.26600000000000001,"5min":0.24399999999999999,"15min":0.22500000000000001},"max":{"1min":38.720999999999997,"5min":38.720999999999997,"15min":38.720999999999997},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.0669999999999999,"15min":0.90200000000000002},"min":{"1min":0.29899999999999999,"5min":0.26500000000000001,"15min":0.25},"max":{"1min":5.3920000000000003,"5min":8.0779999999999994,"15min":8.0779999999999994},"last":0.79200000000000004}]},{"osd":1,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.1850000000000001,"15min":1.1990000000000001},"min":{"1min":0.26800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.2619999999999996,"5min":5.077,"15min":8.9410000000000007},"last":1.254},{"interface":"front","average":{"1min":1.639,"5min":1.1419999999999999,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.25},"max":{"1min":4.7590000000000003,"5min":8.798,"15min":8.798},"last":0.84199999999999997}]},{"osd":2,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.3819999999999999,"15min":1.167},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.5819999999999999,"5min":8.9090000000000007,"15min":8.9090000000000007},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.496,"5min":1.425,"15min":1.236},"min":{"1min":0.316,"5min":0.30499999999999999,"15min":0.29599999999999999},"max":{"1min":4.9790000000000001,"5min":8.8179999999999996,"15min":10.401999999999999},"last":0.96499999999999997}]},{"osd":3,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.2029999999999998,"5min":1.8009999999999999,"15min":1.369},"min":{"1min":0.38700000000000001,"5min":0.31,"15min":0.31},"max":{"1min":16.004000000000001,"5min":16.004000000000001,"15min":16.004000000000001},"last":1.282},{"interface":"front","average":{"1min":1.339,"5min":1.1120000000000001,"15min":1.2470000000000001},"min":{"1min":0.35199999999999998,"5min":0.33900000000000002,"15min":0.32000000000000001},"max":{"1min":4.867,"5min":8.2040000000000006,"15min":8.2040000000000006},"last":1.0129999999999999}]},{"osd":4,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":2.3820000000000001,"5min":1.044,"15min":0.89000000000000001},"min":{"1min":0.33000000000000002,"5min":0.28999999999999998,"15min":0.25900000000000001},"max":{"1min":20.827000000000002,"5min":20.827000000000002,"15min":20.827000000000002},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.7709999999999999,"5min":1.0169999999999999,"15min":0.82899999999999996},"min":{"1min":0.32200000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":10.022,"5min":10.022,"15min":10.022},"last":0.67800000000000005}]},{"osd":5,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":3.8079999999999998,"5min":1.448,"15min":1.073},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.309},"max":{"1min":41.658000000000001,"5min":41.658000000000001,"15min":41.658000000000001},"last":1.069},{"interface":"front","average":{"1min":3.9060000000000001,"5min":1.423,"15min":1.0580000000000001},"min":{"1min":0.39300000000000002,"5min":0.35399999999999998,"15min":0.29199999999999998},"max":{"1min":41.634999999999998,"5min":41.634999999999998,"15min":41.634999999999998},"last":0.75600000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:31:49 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.012,"15min":0.93700000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":4.7960000000000003,"5min":5.2859999999999996,"15min":5.625},"last":1.1739999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.82499999999999996,"15min":0.874},"min":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.318},"max":{"1min":3.835,"5min":4.5359999999999996,"15min":7.0810000000000004},"last":1.2010000000000001}]}]},{"osd":5,"up_from":16,"seq":68719482921,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1661432,"kb_used_data":432,"kb_used_omap":463,"kb_used_meta":1660528,"kb_avail":92710408,"statfs":{"total":96636764160,"available":94935457792,"internally_reserved":0,"allocated":442368,"data_stored":173304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":475086,"internal_metadata":1700380722},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.66300000000000003,"15min":0.73799999999999999},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.23200000000000001},"max":{"1min":5.2619999999999996,"5min":5.2619999999999996,"15min":6.2859999999999996},"last":1.081},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.90700000000000003,"15min":0.84299999999999997},"min":{"1min":0.26400000000000001,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":3.6600000000000001,"5min":5.5810000000000004,"15min":6.4829999999999997},"last":0.99099999999999999}]},{"osd":1,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.0549999999999999,"15min":1.018},"min":{"1min":0.27700000000000002,"5min":0.26600000000000001,"15min":0.26400000000000001},"max":{"1min":5.8109999999999999,"5min":9.2159999999999993,"15min":9.2159999999999993},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.92500000000000004,"15min":0.875},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26200000000000001},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":8.6959999999999997},"last":0.82699999999999996}]},{"osd":2,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.92700000000000005,"15min":0.90200000000000002},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28699999999999998},"max":{"1min":5.0919999999999996,"5min":5.4290000000000003,"15min":6.3460000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.196,"15min":1.145},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":5.8810000000000002,"5min":5.8810000000000002,"15min":6.3540000000000001},"last":1.0429999999999999}]},{"osd":3,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":2.073,"5min":1.181,"15min":1.1160000000000001},"min":{"1min":0.35499999999999998,"5min":0.31,"15min":0.307},"max":{"1min":20.481000000000002,"5min":20.481000000000002,"15min":20.481000000000002},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.25,"15min":1.1870000000000001},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.29099999999999998},"max":{"1min":5.8520000000000003,"5min":5.8520000000000003,"15min":6.444},"last":1.169}]},{"osd":4,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.66900000000000004,"15min":0.68899999999999995},"min":{"1min":0.314,"5min":0.249,"15min":0.249},"max":{"1min":5.4690000000000003,"5min":5.4690000000000003,"15min":6.2149999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79700000000000004,"15min":0.78400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.3490000000000002,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.385}]},{"osd":6,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":1.1319999999999999,"15min":0.98699999999999999},"min":{"1min":0.38700000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":27.823,"5min":27.823,"15min":27.823},"last":1.415},{"interface":"front","average":{"1min":2.5529999999999999,"5min":1.0109999999999999,"15min":0.86199999999999999},"min":{"1min":0.33200000000000002,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":33.023000000000003,"5min":33.023000000000003,"15min":33.023000000000003},"last":1.1439999999999999}]},{"osd":7,"last update":"Sun Dec 17 06:32:01 2023","interfaces":[{"interface":"back","average":{"1min":3.5230000000000001,"5min":1.4139999999999999,"15min":1.0029999999999999},"min":{"1min":0.34000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":42.939,"5min":42.939,"15min":42.939},"last":1.3049999999999999},{"interface":"front","average":{"1min":2.234,"5min":1.0629999999999999,"15min":1.0209999999999999},"min":{"1min":0.44600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":18.196999999999999,"5min":18.196999999999999,"15min":18.196999999999999},"last":1.6319999999999999}]}]},{"osd":4,"up_from":16,"seq":68719482919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1705400,"kb_used_data":432,"kb_used_omap":362,"kb_used_meta":1704597,"kb_avail":92666440,"statfs":{"total":96636764160,"available":94890434560,"internally_reserved":0,"allocated":442368,"data_stored":173304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":371446,"internal_metadata":1745507594},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:34 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":0.64100000000000001,"15min":0.73499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.4829999999999997,"5min":5.4829999999999997,"15min":6.8559999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.93300000000000005,"15min":0.90400000000000003},"min":{"1min":0.26500000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.944,"5min":7.1079999999999997,"15min":8.9420000000000002},"last":1.2609999999999999}]},{"osd":1,"last update":"Sun Dec 17 06:32:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.0129999999999999,"15min":1.0469999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.251},"max":{"1min":4.6609999999999996,"5min":7.3399999999999999,"15min":7.3399999999999999},"last":1.071},{"interface":"front","average":{"1min":1.081,"5min":0.89900000000000002,"15min":1.069},"min":{"1min":0.28899999999999998,"5min":0.27300000000000002,"15min":0.25900000000000001},"max":{"1min":4.0090000000000003,"5min":7.1849999999999996,"15min":9.0129999999999999},"last":0.96999999999999997}]},{"osd":2,"last update":"Sun Dec 17 06:32:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.018,"15min":1.008},"min":{"1min":0.372,"5min":0.308,"15min":0.26700000000000002},"max":{"1min":5.2729999999999997,"5min":7.9189999999999996,"15min":7.9189999999999996},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.03,"5min":1.444,"15min":1.4039999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":5.0099999999999998,"5min":10.103,"15min":10.103},"last":1.3160000000000001}]},{"osd":3,"last update":"Sun Dec 17 06:32:34 2023","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.353,"15min":1.3480000000000001},"min":{"1min":0.34899999999999998,"5min":0.32400000000000001,"15min":0.312},"max":{"1min":6.0069999999999997,"5min":11.417999999999999,"15min":12.627000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.129,"15min":1.218},"min":{"1min":0.35999999999999999,"5min":0.318,"15min":0.318},"max":{"1min":5.9329999999999998,"5min":8.1259999999999994,"15min":10.43},"last":1.131}]},{"osd":5,"last update":"Sun Dec 17 06:32:34 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":0.91200000000000003,"15min":0.81299999999999994},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":13.661,"5min":13.661,"15min":13.661},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.629,"5min":0.83299999999999996,"15min":0.78300000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.26100000000000001},"max":{"1min":17.047000000000001,"5min":17.047000000000001,"15min":17.047000000000001},"last":1.038}]},{"osd":6,"last update":"Sun Dec 17 06:32:34 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.80100000000000005,"15min":0.79800000000000004},"min":{"1min":0.34899999999999998,"5min":0.32400000000000001,"15min":0.27700000000000002},"max":{"1min":3.5030000000000001,"5min":7.5999999999999996,"15min":7.5999999999999996},"last":1},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.0369999999999999,"15min":0.89900000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.35299999999999998},"max":{"1min":3.4590000000000001,"5min":6.4249999999999998,"15min":6.4249999999999998},"last":1.3740000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:32:34 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.33400000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.9990000000000001,"5min":6.1219999999999999,"15min":6.6879999999999997},"last":1.169},{"interface":"front","average":{"1min":1.131,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.39700000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.956,"5min":4.7629999999999999,"15min":6.9379999999999997},"last":1.23}]}]},{"osd":3,"up_from":15,"seq":64424515613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1739832,"kb_used_data":432,"kb_used_omap":132,"kb_used_meta":1739259,"kb_avail":92632008,"statfs":{"total":96636764160,"available":94855176192,"internally_reserved":0,"allocated":442368,"data_stored":173304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":136113,"internal_metadata":1781001295},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.306,"5min":1.1759999999999999,"15min":1.1319999999999999},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.20000000000000001},"max":{"1min":21.396999999999998,"5min":21.396999999999998,"15min":21.396999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.393,"5min":0.93799999999999994,"15min":0.92000000000000004},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":12.132,"5min":12.132,"15min":12.132},"last":1.056}]},{"osd":1,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.052,"5min":1.339,"15min":1.1479999999999999},"min":{"1min":0.28899999999999998,"5min":0.23200000000000001,"15min":0.215},"max":{"1min":29.577000000000002,"5min":29.577000000000002,"15min":29.577000000000002},"last":1.22},{"interface":"front","average":{"1min":2.0870000000000002,"5min":1.4950000000000001,"15min":1.373},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":12.877000000000001,"5min":12.877000000000001,"15min":12.877000000000001},"last":0.67200000000000004}]},{"osd":2,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.1539999999999999,"5min":1.3200000000000001,"15min":1.395},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.253},"max":{"1min":12.920999999999999,"5min":12.920999999999999,"15min":12.920999999999999},"last":1.028},{"interface":"front","average":{"1min":2.4350000000000001,"5min":1.3380000000000001,"15min":1.177},"min":{"1min":0.34999999999999998,"5min":0.29799999999999999,"15min":0.27500000000000002},"max":{"1min":17.669,"5min":17.669,"15min":17.669},"last":0.83599999999999997}]},{"osd":4,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":2.6339999999999999,"5min":1.488,"15min":1.2769999999999999},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":13.095000000000001,"5min":13.228,"15min":13.228},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.9870000000000001,"5min":1.161,"15min":1.0860000000000001},"min":{"1min":0.35899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":12.166,"5min":12.166,"15min":12.166},"last":0.91700000000000004}]},{"osd":5,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.3889999999999998,"5min":1.4710000000000001,"15min":1.234},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34000000000000002},"max":{"1min":29.622,"5min":29.622,"15min":29.622},"last":0.71599999999999997},{"interface":"front","average":{"1min":3.5619999999999998,"5min":1.6699999999999999,"15min":1.409},"min":{"1min":0.38100000000000001,"5min":0.314,"15min":0.314},"max":{"1min":21.038,"5min":21.038,"15min":21.038},"last":0.873}]},{"osd":6,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.1539999999999999,"15min":1.105},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.34999999999999998},"max":{"1min":4.0380000000000003,"5min":5.5220000000000002,"15min":10.683},"last":1.25},{"interface":"front","average":{"1min":1.786,"5min":1.296,"15min":1.1779999999999999},"min":{"1min":0.47299999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":15.619999999999999,"5min":15.619999999999999,"15min":15.619999999999999},"last":1.1679999999999999}]},{"osd":7,"last update":"Sun Dec 17 06:31:59 2023","interfaces":[{"interface":"back","average":{"1min":3.2989999999999999,"5min":1.71,"15min":1.4810000000000001},"min":{"1min":0.42099999999999999,"5min":0.40200000000000002,"15min":0.38},"max":{"1min":18.16,"5min":18.16,"15min":18.16},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.643,"15min":1.288},"min":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.41799999999999998},"max":{"1min":10.282999999999999,"5min":22.334,"15min":22.334},"last":1.117}]}]},{"osd":1,"up_from":15,"seq":64424515618,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1852152,"kb_used_data":432,"kb_used_omap":997,"kb_used_meta":1850714,"kb_avail":92519688,"statfs":{"total":96636764160,"available":94740160512,"internally_reserved":0,"allocated":442368,"data_stored":173304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1021567,"internal_metadata":1895131521},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:29 2023","interfaces":[{"interface":"back","average":{"1min":2.2450000000000001,"5min":1.149,"15min":0.93100000000000005},"min":{"1min":0.218,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":14.401,"5min":14.401,"15min":14.401},"last":0.60699999999999998},{"interface":"front","average":{"1min":3.4630000000000001,"5min":1.6040000000000001,"15min":1.337},"min":{"1min":0.26900000000000002,"5min":0.219,"15min":0.214},"max":{"1min":26.707999999999998,"5min":30.109999999999999,"15min":30.109999999999999},"last":0.85899999999999999}]},{"osd":2,"last update":"Sun Dec 17 06:32:29 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.383,"15min":1.2430000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.224},"max":{"1min":11.439,"5min":25.259,"15min":25.259},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.381,"5min":0.96199999999999997,"15min":1},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":10.406000000000001,"5min":10.406000000000001,"15min":10.406000000000001},"last":1.0229999999999999}]},{"osd":3,"last update":"Sun Dec 17 06:32:29 2023","interfaces":[{"interface":"back","average":{"1min":2.8010000000000002,"5min":1.6240000000000001,"15min":1.3740000000000001},"min":{"1min":0.30299999999999999,"5min":0.249,"15min":0.23400000000000001},"max":{"1min":15.342000000000001,"5min":30.460999999999999,"15min":30.460999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.9409999999999998,"5min":1.7210000000000001,"15min":1.4099999999999999},"min":{"1min":0.317,"5min":0.28999999999999998,"15min":0.25},"max":{"1min":17.106999999999999,"5min":17.106999999999999,"15min":17.106999999999999},"last":0.81299999999999994}]},{"osd":4,"last update":"Sun Dec 17 06:32:29 2023","interfaces":[{"interface":"back","average":{"1min":3.2189999999999999,"5min":1.554,"15min":1.125},"min":{"1min":0.42299999999999999,"5min":0.31,"15min":0.27300000000000002},"max":{"1min":34.561,"5min":34.561,"15min":34.561},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.04,"5min":1.077,"15min":1.071},"min":{"1min":0.38400000000000001,"5min":0.317,"15min":0.30499999999999999},"max":{"1min":3.7650000000000001,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":0.95899999999999996}]},{"osd":5,"last update":"Sun Dec 17 06:32:29 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.1970000000000001,"15min":1.0249999999999999},"min":{"1min":0.36699999999999999,"5min":0.34599999999999997,"15min":0.315},"max":{"1min":10.449,"5min":10.449,"15min":10.449},"last":1.1850000000000001},{"interface":"front","average":{"1min":2.351,"5min":1.4970000000000001,"15min":1.302},"min":{"1min":0.371,"5min":0.37,"15min":0.32600000000000001},"max":{"1min":14.446,"5min":30.277999999999999,"15min":30.277999999999999},"last":1.002}]},{"osd":6,"last update":"Sun Dec 17 06:32:29 2023","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.196,"15min":1.1970000000000001},"min":{"1min":0.40999999999999998,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":3.7320000000000002,"5min":5.2839999999999998,"15min":8.8840000000000003},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.575,"15min":1.1950000000000001},"min":{"1min":0.49099999999999999,"5min":0.39100000000000001,"15min":0.38700000000000001},"max":{"1min":10.475,"5min":19.016999999999999,"15min":19.016999999999999},"last":1.2989999999999999}]},{"osd":7,"last update":"Sun Dec 17 06:32:29 2023","interfaces":[{"interface":"back","average":{"1min":2.7469999999999999,"5min":1.6439999999999999,"15min":1.377},"min":{"1min":0.47199999999999998,"5min":0.40699999999999997,"15min":0.378},"max":{"1min":14.489000000000001,"5min":30.242999999999999,"15min":30.242999999999999},"last":1.153},{"interface":"front","average":{"1min":1.466,"5min":1.345,"15min":1.292},"min":{"1min":0.51300000000000001,"5min":0.42299999999999999,"15min":0.378},"max":{"1min":6.0090000000000003,"5min":6.0279999999999996,"15min":8.9589999999999996},"last":1.2609999999999999}]}]},{"osd":0,"up_from":15,"seq":64424515620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1399612,"kb_used_data":1012,"kb_used_omap":422,"kb_used_meta":1398169,"kb_avail":92972228,"statfs":{"total":96636764160,"available":95203561472,"internally_reserved":0,"allocated":1036288,"data_stored":763672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":432402,"internal_metadata":1431725806},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":0.88400000000000001,"15min":0.97199999999999998},"min":{"1min":0.20200000000000001,"5min":0.184,"15min":0.184},"max":{"1min":8.3789999999999996,"5min":8.9369999999999994,"15min":8.9369999999999994},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.96199999999999997,"15min":1.181},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1079999999999997,"5min":5.1470000000000002,"15min":8.1039999999999992},"last":0.90400000000000003}]},{"osd":2,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.159,"15min":1.194},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.2759999999999998,"5min":7.2759999999999998,"15min":10.146000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.2130000000000001,"15min":1.2889999999999999},"min":{"1min":0.26600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.4770000000000003,"5min":13.676,"15min":13.676},"last":0.78400000000000003}]},{"osd":3,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":1.2929999999999999,"15min":1.2749999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.25},"max":{"1min":2.5680000000000001,"5min":13.721,"15min":13.721},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.0840000000000001,"15min":1.1870000000000001},"min":{"1min":0.37,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":6.2089999999999996,"5min":8.0570000000000004,"15min":10.037000000000001},"last":0.82899999999999996}]},{"osd":4,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.145,"15min":1.1919999999999999},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.0579999999999998,"5min":13.755000000000001,"15min":13.755000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.0389999999999999,"15min":0.93200000000000005},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.29199999999999998},"max":{"1min":6.4649999999999999,"5min":6.4649999999999999,"15min":7.3540000000000001},"last":1.2729999999999999}]},{"osd":5,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":0.95499999999999996,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.32700000000000001},"max":{"1min":6.5090000000000003,"5min":6.5090000000000003,"15min":7.4349999999999996},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.0089999999999999,"15min":1.149},"min":{"1min":0.379,"5min":0.31,"15min":0.31},"max":{"1min":4.1589999999999998,"5min":5.101,"15min":8.1820000000000004},"last":1.0409999999999999}]},{"osd":6,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.25,"15min":1.3069999999999999},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.34300000000000003},"max":{"1min":6.202,"5min":13.819000000000001,"15min":13.819000000000001},"last":1.117},{"interface":"front","average":{"1min":2.1549999999999998,"5min":1.1499999999999999,"15min":1.087},"min":{"1min":0.38,"5min":0.38,"15min":0.378},"max":{"1min":11.897,"5min":11.897,"15min":11.897},"last":1.3540000000000001}]},{"osd":7,"last update":"Sun Dec 17 06:31:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.087,"15min":1.2030000000000001},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38500000000000001},"max":{"1min":4.2119999999999997,"5min":5.1760000000000002,"15min":8.2110000000000003},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.266,"15min":1.4059999999999999},"min":{"1min":0.39500000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.242,"5min":13.786,"15min":13.786},"last":1.2150000000000001}]}]},{"osd":2,"up_from":14,"seq":60129548321,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1295032,"kb_used_data":432,"kb_used_omap":313,"kb_used_meta":1294278,"kb_avail":93076808,"statfs":{"total":96636764160,"available":95310651392,"internally_reserved":0,"allocated":442368,"data_stored":173304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":320629,"internal_metadata":1325341579},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.071,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.3380000000000001,"5min":7.5919999999999996,"15min":8.1500000000000004},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.22,"15min":1.125},"min":{"1min":0.22,"5min":0.217,"15min":0.20599999999999999},"max":{"1min":4.0030000000000001,"5min":10.301,"15min":10.301},"last":0.746}]},{"osd":1,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.98099999999999998,"15min":1.0920000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.048,"5min":6.8010000000000002,"15min":7.7320000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.218,"15min":1.2370000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.9859999999999998,"5min":10.345000000000001,"15min":10.345000000000001},"last":1.016}]},{"osd":3,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.482,"15min":1.411},"min":{"1min":0.27400000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.3250000000000002,"5min":10.706,"15min":10.706},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.216,"15min":1.1899999999999999},"min":{"1min":0.32900000000000001,"5min":0.29699999999999999,"15min":0.26100000000000001},"max":{"1min":14.513999999999999,"5min":14.513999999999999,"15min":14.513999999999999},"last":1.103}]},{"osd":4,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.073,"15min":1.073},"min":{"1min":0.39000000000000001,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":3.1899999999999999,"5min":6.1870000000000003,"15min":8.0860000000000003},"last":1.292},{"interface":"front","average":{"1min":1.399,"5min":1.276,"15min":1.1679999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":10.583,"5min":10.583,"15min":10.583},"last":1.24}]},{"osd":5,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":1.6859999999999999,"15min":1.383},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":30.448,"5min":30.448,"15min":30.448},"last":1.4510000000000001},{"interface":"front","average":{"1min":2.6619999999999999,"5min":1.6459999999999999,"15min":1.4610000000000001},"min":{"1min":0.38500000000000001,"5min":0.374,"15min":0.33900000000000002},"max":{"1min":21.797999999999998,"5min":21.797999999999998,"15min":21.797999999999998},"last":1.5600000000000001}]},{"osd":6,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.758,"15min":1.5089999999999999},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.376},"max":{"1min":15.653,"5min":15.653,"15min":15.653},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.329,"15min":1.2290000000000001},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.35999999999999999},"max":{"1min":15.82,"5min":15.82,"15min":15.82},"last":1.3959999999999999}]},{"osd":7,"last update":"Sun Dec 17 06:32:19 2023","interfaces":[{"interface":"back","average":{"1min":1.728,"5min":1.4770000000000001,"15min":1.335},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.37},"max":{"1min":8.0250000000000004,"5min":9.5489999999999995,"15min":9.5489999999999995},"last":1.359},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.508,"15min":1.47},"min":{"1min":0.44500000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"max":{"1min":6.431,"5min":7.9429999999999996,"15min":10.446},"last":1.4830000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-17T06:32:46.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-17T06:32:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e2e20640 1 Processor -- start 2023-12-17T06:32:46.204 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e2e20640 1 -- start start 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e2e20640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 0x7ff4dc0cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e2e20640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7ff4dc0d0120 0x7ff4dc0b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e2e20640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4dc0d0570 con 0x7ff4dc15d2b0 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e2e20640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4dc0aff30 con 0x7ff4dc0d0120 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e2e20640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0d1ec0 con 0x7ff4dc15d690 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e0bbc640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4dc15d2b0 0x7ff4dc0d17b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45594/0 (socket says 172.21.15.28:45594) 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e0bbc640 1 -- 172.21.15.28:0/3537823021 learned_addr learned my addr 172.21.15.28:0/3537823021 (peer_addr_for_me v1:172.21.15.28:0/0) 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779645784 0 0) 0x7ff4cc001120 con 0x7ff4dc15d2b0 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c4003530 con 0x7ff4dc15d2b0 2023-12-17T06:32:46.207 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4dbfff640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 0x7ff4dc0cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e13bd640 1 --2- 172.21.15.28:0/3537823021 >> v2:172.21.15.122:3300/0 conn(0x7ff4dc0d0120 0x7ff4dc0b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1076066853 0 0) 0x7ff4c4003530 con 0x7ff4dc15d2b0 2023-12-17T06:32:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 msgr2=0x7ff4dc0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 0x7ff4dc0cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 >> v2:172.21.15.122:3300/0 conn(0x7ff4dc0d0120 msgr2=0x7ff4dc0b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 --2- 172.21.15.28:0/3537823021 >> v2:172.21.15.122:3300/0 conn(0x7ff4dc0d0120 0x7ff4dc0b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc0b01d0 con 0x7ff4dc15d2b0 2023-12-17T06:32:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4124223854 0 0) 0x7ff4cc001810 con 0x7ff4dc15d2b0 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 >> v1:172.21.15.28:6789/0 conn(0x7ff4dc15d2b0 legacy=0x7ff4dc0d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c4003980 0x7ff4c4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c40047d0 0x7ff4c4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 0x7ff4c4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0aff30 con 0x7ff4c4003980 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0d1ec0 con 0x7ff4c40047d0 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c400ba40 con 0x7ff4dc15d690 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e0bbc640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c4003980 0x7ff4c4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.209 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e13bd640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 0x7ff4c4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4dbfff640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c40047d0 0x7ff4c4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e0bbc640 1 -- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 msgr2=0x7ff4c4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e0bbc640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 0x7ff4c4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e0bbc640 1 -- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c40047d0 msgr2=0x7ff4c4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e0bbc640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c40047d0 0x7ff4c4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e0bbc640 1 -- 172.21.15.28:0/3537823021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c4003530 con 0x7ff4c4003980 2023-12-17T06:32:46.210 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.197+0000 7ff4e13bd640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc15d690 0x7ff4c4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-17T06:32:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e0bbc640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c4003980 0x7ff4c4003da0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc001060 tx=0x7ff4cc01cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11cc6be5404df4c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc0058e0 con 0x7ff4c4003980 2023-12-17T06:32:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4cc005a60 con 0x7ff4c4003980 2023-12-17T06:32:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4db7fe640 1 -- 172.21.15.28:0/3537823021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc005e10 con 0x7ff4c4003980 2023-12-17T06:32:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e2e20640 1 -- 172.21.15.28:0/3537823021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c4003980 msgr2=0x7ff4c4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e2e20640 1 --2- 172.21.15.28:0/3537823021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c4003980 0x7ff4c4003da0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc001060 tx=0x7ff4cc01cbb0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e13bd640 1 -- 172.21.15.28:0/3537823021 reap_dead start 2023-12-17T06:32:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e2e20640 1 -- 172.21.15.28:0/3537823021 shutdown_connections 2023-12-17T06:32:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e2e20640 1 -- 172.21.15.28:0/3537823021 >> 172.21.15.28:0/3537823021 conn(0x7ff4dc09c510 msgr2=0x7ff4dc0d31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e2e20640 1 -- 172.21.15.28:0/3537823021 shutdown_connections 2023-12-17T06:32:46.214 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e2e20640 1 -- 172.21.15.28:0/3537823021 wait complete. 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e2e20640 1 Processor -- start 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.201+0000 7ff4e2e20640 1 -- start start 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c4003980 0x7ff4dc157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc0b7ee0 0x7ff4dc06c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc0b8360 0x7ff4dc06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc157c50 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0b5f20 con 0x7ff4c4003980 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc0b6200 con 0x7ff4dc0b8360 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc0b7ee0 0x7ff4dc06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc0b7ee0 0x7ff4dc06c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33568/0 (socket says 172.21.15.28:33568) 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 -- 172.21.15.28:0/1351388947 learned_addr learned my addr 172.21.15.28:0/1351388947 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 -- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc0b8360 msgr2=0x7ff4dc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc0b8360 0x7ff4dc06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 -- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c4003980 msgr2=0x7ff4dc157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c4003980 0x7ff4dc157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 -- 172.21.15.28:0/1351388947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc0b64e0 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4dbfff640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc0b7ee0 0x7ff4dc06c490 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000f580 tx=0x7ff4d000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4d97fa640 1 -- 172.21.15.28:0/1351388947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0014870 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4d97fa640 1 -- 172.21.15.28:0/1351388947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d00149d0 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4dc0b67c0 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4d97fa640 1 -- 172.21.15.28:0/1351388947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0014d60 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc0b7a40 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc0679f0 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4d97fa640 1 -- 172.21.15.28:0/1351388947 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7ff4d0015410 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4d97fa640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff4c8066470 0x7ff4c8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4d97fa640 1 -- 172.21.15.28:0/1351388947 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff4d0084540 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.205+0000 7ff4e0bbc640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff4c8066470 0x7ff4c8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.209+0000 7ff4e0bbc640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff4c8066470 0x7ff4c8068930 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0aa000 tx=0x7ff4cc000ff0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.209+0000 7ff4d97fa640 1 -- 172.21.15.28:0/1351388947 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7ff4d0051cd0 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.369 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.361+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7ff4dc0693d0 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.375 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4d97fa640 1 -- 172.21.15.28:0/1351388947 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) v1 ==== 123+0+0 (secure 0 0 0) 0x7ff4d00552f0 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.377 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4d97fa640 1 -- 172.21.15.28:0/1351388947 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff4d0051cd0 con 0x7ff4dc0b7ee0 2023-12-17T06:32:46.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff4c8066470 msgr2=0x7ff4c8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff4c8066470 0x7ff4c8068930 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0aa000 tx=0x7ff4cc000ff0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc0b7ee0 msgr2=0x7ff4dc06c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc0b7ee0 0x7ff4dc06c490 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000f580 tx=0x7ff4d000fa40 comp rx=0 tx=0).stop 2023-12-17T06:32:46.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 shutdown_connections 2023-12-17T06:32:46.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7ff4c8066470 0x7ff4c8068930 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.379 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7ff4dc0b8360 0x7ff4dc06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4dc0b7ee0 0x7ff4dc06c490 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 --2- 172.21.15.28:0/1351388947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4c4003980 0x7ff4dc157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 >> 172.21.15.28:0/1351388947 conn(0x7ff4dc09c510 msgr2=0x7ff4dc0d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:46.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 shutdown_connections 2023-12-17T06:32:46.380 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.369+0000 7ff4e2e20640 1 -- 172.21.15.28:0/1351388947 wait complete. 2023-12-17T06:32:46.395 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-17T06:32:46.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-17T06:32:46.511 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbf4bd640 1 Processor -- start 2023-12-17T06:32:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbf4bd640 1 -- start start 2023-12-17T06:32:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbf4bd640 1 --2- >> v2:172.21.15.122:3300/0 conn(0x7f0db815aee0 0x7f0db815f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbf4bd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0db80ce6f0 0x7f0db8070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbf4bd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db815fbe0 con 0x7f0db815b2c0 2023-12-17T06:32:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbf4bd640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80aee80 con 0x7f0db80ce6f0 2023-12-17T06:32:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbf4bd640 1 -- --> v2:172.21.15.122:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80cec40 con 0x7f0db815aee0 2023-12-17T06:32:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbda5a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0db80ce6f0 0x7f0db8070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbda5a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0db80ce6f0 0x7f0db8070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33576/0 (socket says 172.21.15.28:33576) 2023-12-17T06:32:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbda5a640 1 -- 172.21.15.28:0/2191851906 learned_addr learned my addr 172.21.15.28:0/2191851906 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbd259640 1 --2- 172.21.15.28:0/2191851906 >> v2:172.21.15.122:3300/0 conn(0x7f0db815aee0 0x7f0db815f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbda5a640 1 -- 172.21.15.28:0/2191851906 >> v2:172.21.15.122:3300/0 conn(0x7f0db815aee0 msgr2=0x7f0db815f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbda5a640 1 --2- 172.21.15.28:0/2191851906 >> v2:172.21.15.122:3300/0 conn(0x7f0db815aee0 0x7f0db815f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbda5a640 1 -- 172.21.15.28:0/2191851906 >> v1:172.21.15.122:6789/0 conn(0x7f0db815b2c0 legacy=0x7f0db806a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-17T06:32:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.505+0000 7f0dbda5a640 1 -- 172.21.15.28:0/2191851906 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80cef20 con 0x7f0db80ce6f0 2023-12-17T06:32:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 --2- 172.21.15.28:0/2191851906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db80ce6f0 0x7f0db8070cb0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f0db80d03b0 tx=0x7f0db001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e149a852d0709892 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0daffff640 1 -- 172.21.15.28:0/2191851906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db00a56a0 con 0x7f0db80ce6f0 2023-12-17T06:32:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0daffff640 1 -- 172.21.15.28:0/2191851906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0db00a5800 con 0x7f0db80ce6f0 2023-12-17T06:32:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0daffff640 1 -- 172.21.15.28:0/2191851906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db00a5bb0 con 0x7f0db80ce6f0 2023-12-17T06:32:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2191851906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db80ce6f0 msgr2=0x7f0db8070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2191851906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db80ce6f0 0x7f0db8070cb0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f0db80d03b0 tx=0x7f0db001c8d0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2191851906 shutdown_connections 2023-12-17T06:32:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2191851906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db80ce6f0 0x7f0db8070cb0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2191851906 >> v2:172.21.15.122:3300/0 conn(0x7f0db815aee0 0x7f0db815f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2191851906 >> 172.21.15.28:0/2191851906 conn(0x7f0db80cf5d0 msgr2=0x7f0db80cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2191851906 shutdown_connections 2023-12-17T06:32:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2191851906 wait complete. 2023-12-17T06:32:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 Processor -- start 2023-12-17T06:32:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- start start 2023-12-17T06:32:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 --2- >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0db80ce6f0 0x7f0db814f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0db815aee0 0x7f0db81460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db815b2c0 0x7f0db815a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db814c100 con 0x7f0db815b2c0 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8146620 con 0x7f0db815aee0 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbf4bd640 1 -- --> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db814c350 con 0x7f0db80ce6f0 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db815b2c0 0x7f0db815a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db815b2c0 0x7f0db815a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33584/0 (socket says 172.21.15.28:33584) 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 -- 172.21.15.28:0/2797844463 learned_addr learned my addr 172.21.15.28:0/2797844463 (peer_addr_for_me v2:172.21.15.28:0/0) 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbd259640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0db80ce6f0 0x7f0db814f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbca58640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0db815aee0 0x7f0db81460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 -- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0db80ce6f0 msgr2=0x7f0db814f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0db80ce6f0 0x7f0db814f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 -- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0db815aee0 msgr2=0x7f0db81460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0db815aee0 0x7f0db81460e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 -- 172.21.15.28:0/2797844463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0009b30 con 0x7f0db815b2c0 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dbda5a640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db815b2c0 0x7f0db815a0c0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f0db00af040 tx=0x7f0db0019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.509+0000 7f0dadffb640 1 -- 172.21.15.28:0/2797844463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db00b8670 con 0x7f0db815b2c0 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dadffb640 1 -- 172.21.15.28:0/2797844463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f0db00b87d0 con 0x7f0db815b2c0 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db814c630 con 0x7f0db815b2c0 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dadffb640 1 -- 172.21.15.28:0/2797844463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db00b8b60 con 0x7f0db815b2c0 2023-12-17T06:32:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db814cbf0 con 0x7f0db815b2c0 2023-12-17T06:32:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db80674d0 con 0x7f0db815b2c0 2023-12-17T06:32:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dadffb640 1 -- 172.21.15.28:0/2797844463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 85854+0+0 (secure 0 0 0) 0x7f0db00b8cc0 con 0x7f0db815b2c0 2023-12-17T06:32:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dadffb640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0d9c066470 0x7f0d9c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-17T06:32:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dadffb640 1 -- 172.21.15.28:0/2797844463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0db011f130 con 0x7f0db815b2c0 2023-12-17T06:32:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dbd259640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0d9c066470 0x7f0d9c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-17T06:32:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.513+0000 7f0dbd259640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0d9c066470 0x7f0d9c068930 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0db400a970 tx=0x7f0db4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-17T06:32:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.517+0000 7f0dadffb640 1 -- 172.21.15.28:0/2797844463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172666 (secure 0 0 0) 0x7f0db00ec940 con 0x7f0db815b2c0 2023-12-17T06:32:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.705+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0db815fbe0 con 0x7f0db815b2c0 2023-12-17T06:32:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.705+0000 7f0dadffb640 1 -- 172.21.15.28:0/2797844463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3703 (secure 0 0 0) 0x7f0db815fbe0 con 0x7f0db815b2c0 2023-12-17T06:32:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: 2023-12-17T06:32:46.714 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4682,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/992095456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":992095456},{"type":"v1","addr":"172.21.15.122:6835","nonce":992095456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4685,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/3174619236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":3174619236},{"type":"v1","addr":"172.21.15.122:6837","nonce":3174619236}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4691,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3342817962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3342817962},{"type":"v1","addr":"172.21.15.28:6835","nonce":3342817962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":10143,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1647350561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1647350561},{"type":"v1","addr":"172.21.15.28:6833","nonce":1647350561}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2023-12-17T06:32:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2023-12-17T06:32:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0d9c066470 msgr2=0x7f0d9c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0d9c066470 0x7f0d9c068930 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f0db400a970 tx=0x7f0db4008040 comp rx=0 tx=0).stop 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db815b2c0 msgr2=0x7f0db815a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db815b2c0 0x7f0db815a0c0 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f0db00af040 tx=0x7f0db0019670 comp rx=0 tx=0).stop 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 shutdown_connections 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:6832/3228142796,v1:172.21.15.122:6833/3228142796] conn(0x7f0d9c066470 0x7f0d9c068930 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db815b2c0 0x7f0db815a0c0 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0db815aee0 0x7f0db81460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 --2- 172.21.15.28:0/2797844463 >> [v2:172.21.15.122:3301/0,v1:172.21.15.122:6790/0] conn(0x7f0db80ce6f0 0x7f0db814f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 >> 172.21.15.28:0/2797844463 conn(0x7f0db80cf5d0 msgr2=0x7f0db806b540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 shutdown_connections 2023-12-17T06:32:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2023-12-17T06:32:46.709+0000 7f0dbf4bd640 1 -- 172.21.15.28:0/2797844463 wait complete. 2023-12-17T06:32:46.733 DEBUG:tasks.cephfs.filesystem:already dead... 2023-12-17T06:32:46.733 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-17T06:32:46.734 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-17T06:32:46.734 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:32:46.745 INFO:tasks.ceph.mds.a:Stopped 2023-12-17T06:32:46.745 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-17T06:32:46.745 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:32:46.800 INFO:tasks.ceph.mds.c:Stopped 2023-12-17T06:32:46.801 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-17T06:32:46.801 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:32:46.823 INFO:tasks.ceph.mds.b:Stopped 2023-12-17T06:32:46.823 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-17T06:32:46.823 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:32:46.887 INFO:tasks.ceph.mds.d:Stopped 2023-12-17T06:32:46.887 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-17T06:32:46.887 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-17T06:32:46.887 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:32:48.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-12-17T06:32:52.989 INFO:tasks.ceph.osd.0:Stopped 2023-12-17T06:32:52.990 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-17T06:32:52.990 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:32:54.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-17T06:32:54.672 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-12-17T06:32:59.092 INFO:tasks.ceph.osd.1:Stopped 2023-12-17T06:32:59.093 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-17T06:32:59.093 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:00.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-17T06:33:00.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-17T06:33:05.195 INFO:tasks.ceph.osd.2:Stopped 2023-12-17T06:33:05.196 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-17T06:33:05.196 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:06.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-12-17T06:33:06.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-17T06:33:11.298 INFO:tasks.ceph.osd.3:Stopped 2023-12-17T06:33:11.298 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-17T06:33:11.298 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:12.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-12-17T06:33:12.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-12-17T06:33:17.400 INFO:tasks.ceph.osd.4:Stopped 2023-12-17T06:33:17.400 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-17T06:33:17.400 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:18.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2023-12-17T06:33:23.502 INFO:tasks.ceph.osd.5:Stopped 2023-12-17T06:33:23.503 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-17T06:33:23.503 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:23.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-12-17T06:33:23.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-12-17T06:33:29.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-12-17T06:33:29.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-12-17T06:33:29.605 INFO:tasks.ceph.osd.6:Stopped 2023-12-17T06:33:29.606 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-17T06:33:29.606 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:35.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-12-17T06:33:35.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-12-17T06:33:35.708 INFO:tasks.ceph.osd.7:Stopped 2023-12-17T06:33:35.708 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-17T06:33:35.709 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-17T06:33:35.709 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:35.763 INFO:tasks.ceph.mgr.y:Stopped 2023-12-17T06:33:35.763 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-17T06:33:35.763 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:35.797 INFO:tasks.ceph.mgr.x:Stopped 2023-12-17T06:33:35.797 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-17T06:33:35.797 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-17T06:33:35.797 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:35.837 INFO:tasks.ceph.mon.a:Stopped 2023-12-17T06:33:35.838 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-17T06:33:35.838 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:35.881 INFO:tasks.ceph.mon.b:Stopped 2023-12-17T06:33:35.881 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-17T06:33:35.881 INFO:teuthology.orchestra.run:waiting for 300 2023-12-17T06:33:35.915 INFO:tasks.ceph.mon.c:Stopped 2023-12-17T06:33:35.915 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-17T06:33:35.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-17T06:33:36.032 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2023-12-17T06:33:36.032 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-17T06:33:36.213 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2023-12-17T06:33:36.213 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-17T06:33:36.477 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi028.front.sepia.ceph.com 2023-12-17T06:33:36.477 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-17T06:33:36.872 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi028.front.sepia.ceph.com 2023-12-17T06:33:36.872 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-17T06:33:37.111 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi122.front.sepia.ceph.com 2023-12-17T06:33:37.111 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-17T06:33:37.409 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi122.front.sepia.ceph.com 2023-12-17T06:33:37.409 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-17T06:33:37.614 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi122.front.sepia.ceph.com 2023-12-17T06:33:37.614 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-17T06:33:37.793 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi122.front.sepia.ceph.com 2023-12-17T06:33:37.794 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-17T06:33:37.971 INFO:tasks.ceph:Archiving mon data... 2023-12-17T06:33:37.972 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888/data/mon.a.tgz 2023-12-17T06:33:37.973 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2023-12-17T06:33:37.979 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.Fj9Ik9Oztk 2023-12-17T06:33:37.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Fj9Ik9Oztk 2023-12-17T06:33:38.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.Fj9Ik9Oztk 2023-12-17T06:33:38.344 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.Fj9Ik9Oztk is 609KB 2023-12-17T06:33:38.410 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.Fj9Ik9Oztk 2023-12-17T06:33:38.416 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888/data/mon.b.tgz 2023-12-17T06:33:38.416 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2023-12-17T06:33:38.423 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.GTOQeI0ULP 2023-12-17T06:33:38.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.GTOQeI0ULP 2023-12-17T06:33:38.593 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.GTOQeI0ULP 2023-12-17T06:33:38.688 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.GTOQeI0ULP is 615KB 2023-12-17T06:33:38.753 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.GTOQeI0ULP 2023-12-17T06:33:38.760 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888/data/mon.c.tgz 2023-12-17T06:33:38.760 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2023-12-17T06:33:38.807 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.CP5FEtEpKP 2023-12-17T06:33:38.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.CP5FEtEpKP 2023-12-17T06:33:38.959 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.CP5FEtEpKP 2023-12-17T06:33:39.024 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.CP5FEtEpKP is 615KB 2023-12-17T06:33:39.089 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.CP5FEtEpKP 2023-12-17T06:33:39.095 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-17T06:33:39.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-17T06:33:39.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-17T06:33:39.218 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-17T06:33:39.230 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-17T06:33:39.230 INFO:tasks.ceph:Archiving crash dumps... 2023-12-17T06:33:39.232 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888/remote/smithi028/crash 2023-12-17T06:33:39.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-17T06:33:39.277 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888/remote/smithi122/crash 2023-12-17T06:33:39.277 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-17T06:33:39.291 INFO:tasks.ceph:Shutting down logrotate 2023-12-17T06:33:39.292 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 257, in ceph_log logrotater.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 222, in end self.thread.get() File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 189, in invoke_logrotate run.wait(procs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-12-17T06:33:39.306 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-17T06:33:39.318 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 257, in ceph_log logrotater.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 222, in end self.thread.get() File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/ceph.py", line 189, in invoke_logrotate run.wait(procs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi028 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-12-17T06:33:39.322 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-17T06:33:39.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-17T06:33:39.339 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-17T06:33:39.451 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-17T06:33:39.460 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-17T06:33:39.460 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-17T06:33:39.462 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-17T06:33:39.523 INFO:teuthology.orchestra.run.smithi122.stdout: remote refid st t when poll reach delay offset jitter 2023-12-17T06:33:39.523 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================== 2023-12-17T06:33:39.523 INFO:teuthology.orchestra.run.smithi122.stdout:*hv01.front.sepi 44.4.53.2 2 u 40 1024 377 0.156 +0.663 0.916 2023-12-17T06:33:39.523 INFO:teuthology.orchestra.run.smithi122.stdout:+hv02.front.sepi 63.231.80.2 3 u 127 1024 377 0.144 +2.079 1.173 2023-12-17T06:33:39.523 INFO:teuthology.orchestra.run.smithi122.stdout:+hv03.front.sepi 74.6.168.72 3 u 266 1024 377 0.136 +0.765 1.054 2023-12-17T06:33:39.524 INFO:teuthology.orchestra.run.smithi122.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-12-17T06:33:39.532 INFO:teuthology.orchestra.run.smithi028.stdout: remote refid st t when poll reach delay offset jitter 2023-12-17T06:33:39.533 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================== 2023-12-17T06:33:39.533 INFO:teuthology.orchestra.run.smithi028.stdout:*hv01.front.sepi 44.4.53.2 2 u 513 1024 377 0.148 +1.338 1.361 2023-12-17T06:33:39.533 INFO:teuthology.orchestra.run.smithi028.stdout:+hv02.front.sepi 63.231.80.2 3 u 376 1024 377 0.146 +2.741 1.624 2023-12-17T06:33:39.533 INFO:teuthology.orchestra.run.smithi028.stdout:+hv03.front.sepi 74.6.168.72 3 u 335 1024 377 0.163 +1.374 1.387 2023-12-17T06:33:39.533 INFO:teuthology.orchestra.run.smithi028.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-12-17T06:33:39.534 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-17T06:33:39.544 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-17T06:33:39.544 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-17T06:33:39.554 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-17T06:33:39.562 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-17T06:33:39.571 INFO:teuthology.task.internal:Duration was 31651.627478 seconds 2023-12-17T06:33:39.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-17T06:33:39.580 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-17T06:33:39.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-17T06:33:39.583 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-17T06:33:39.728 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-17T06:33:39.728 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2023-12-17T06:33:39.728 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-17T06:33:39.743 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2023-12-17T06:33:39.743 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-17T06:33:39.754 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-17T06:33:39.754 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-17T06:33:39.788 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-17T06:33:39.888 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-17T06:33:39.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-17T06:33:39.891 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-17T06:33:40.025 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-12-17T06:33:40.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-17T06:33:40.539 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-17T06:33:40.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-17T06:33:40.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-17T06:33:40.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-17T06:33:40.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-17T06:33:40.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-17T06:33:40.645 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2023-12-17T06:33:40.671 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2023-12-17T06:33:40.686 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-17T06:33:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T06:33:40.719 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-17T06:33:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-17T06:33:40.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-17T06:33:40.741 INFO:teuthology.task.internal:Transferring archived files... 2023-12-17T06:33:40.742 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888/remote/smithi028 2023-12-17T06:33:40.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-17T06:33:40.788 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888/remote/smithi122 2023-12-17T06:33:40.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-17T06:33:40.807 INFO:teuthology.task.internal:Removing archive directory... 2023-12-17T06:33:40.807 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-17T06:33:40.825 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-17T06:33:40.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-17T06:33:40.862 INFO:teuthology.task.internal:Not uploading archives. 2023-12-17T06:33:40.863 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-17T06:33:40.872 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-17T06:33:40.873 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-17T06:33:40.875 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-17T06:33:40.879 INFO:teuthology.orchestra.run.smithi028.stdout: 655473 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 17 06:33 /home/ubuntu/cephtest 2023-12-17T06:33:40.899 INFO:teuthology.orchestra.run.smithi122.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 17 06:33 /home/ubuntu/cephtest 2023-12-17T06:33:40.901 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-17T06:33:40.911 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-17T06:33:40.923 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi028 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-12-17T06:33:40.923 INFO:teuthology.nuke:Checking targets against current locks 2023-12-17T06:33:40.951 DEBUG:teuthology.nuke:shortname: smithi028 2023-12-17T06:33:40.952 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-17T06:33:40.978 DEBUG:teuthology.nuke:shortname: smithi122 2023-12-17T06:33:40.978 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-17T06:33:40.982 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 21:35:49.950107', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHqnBEcUc0TyMy+JYxM1Vd8gvT1HQU6hovtByw5v0ToGSNJ0XNATQJQEQsRRZ3thjojEItyZw/kqsayZ20D4rSw='} 2023-12-17T06:33:41.007 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494888', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 21:35:49.948581', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEJJ6URhJF7Vp3S0yxJdaNa9SgfnASuuzK9ws0P2U9Q88JHdZmDOM9uOsQdZ8pzetsGdReJKRBENC7VFvkslvzw='} 2023-12-17T06:33:41.035 INFO:teuthology.orchestra.console.smithi028:Power off 2023-12-17T06:33:41.035 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-17T06:33:41.062 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2023-12-17T06:33:41.062 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-17T06:33:41.078 INFO:teuthology.orchestra.console.smithi122:Power off 2023-12-17T06:33:41.079 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-17T06:33:41.089 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2023-12-17T06:33:41.102 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2023-12-17T06:33:41.102 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-17T06:33:41.127 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2023-12-17T06:33:45.090 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-17T06:33:45.128 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-17T06:33:45.201 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2023-12-17T06:33:45.237 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2023-12-17T06:33:49.207 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-17T06:33:49.238 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-17T06:33:49.316 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2023-12-17T06:33:49.346 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2023-12-17T06:33:49.417 INFO:teuthology.orchestra.console.smithi028:Power off completed 2023-12-17T06:33:49.447 INFO:teuthology.orchestra.console.smithi122:Power off completed 2023-12-17T06:33:49.551 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} duration: 31651.627477645874 failure_reason: 'Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4f1ecda474a04a72955af7947be6cb51 status: fail success: false 2023-12-17T06:33:49.552 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-17T06:33:49.636 INFO:teuthology.run:FAIL